Jul 01 18:04:06.656534 ip-10-0-132-161 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 01 18:04:06.656544 ip-10-0-132-161 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 01 18:04:06.656551 ip-10-0-132-161 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 01 18:04:06.656782 ip-10-0-132-161 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 01 18:04:17.817771 ip-10-0-132-161 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 01 18:04:17.817788 ip-10-0-132-161 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6adc8d157dad4756a6e8dd1ae2bcedcc --
Jul 01 18:06:18.333314 ip-10-0-132-161 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 18:06:18.879845 ip-10-0-132-161 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:06:18.879845 ip-10-0-132-161 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 18:06:18.879845 ip-10-0-132-161 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:06:18.879845 ip-10-0-132-161 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 18:06:18.879845 ip-10-0-132-161 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:06:18.879845 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.671034    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 18:06:18.879845 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675027    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:06:18.879845 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675037    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:06:18.879845 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675040    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:06:18.879845 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675043    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:06:18.879845 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675046    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:06:18.879845 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675049    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:06:18.879845 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675052    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:06:18.879845 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675055    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:06:18.738951 ip-10-0-132-161 systemd[1]: Started Kubernetes Kubelet.
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675058    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675061    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675064    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675066    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675069    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675071    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675074    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675076    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675079    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675082    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675084    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675087    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675090    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675093    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675095    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675101    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675106    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675110    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:06:18.895373 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675114    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675118    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675121    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675124    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675127    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675130    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675133    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675136    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675139    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675142    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675145    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675147    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675151    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675153    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675156    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675158    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675161    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675163    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675166    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675169    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:06:18.896440 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675172    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675174    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675177    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675181    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675183    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675186    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675188    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675191    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675194    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675196    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675199    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675202    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675204    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675207    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675210    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675213    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675215    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675218    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675220    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675223    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:06:18.897203 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675225    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675228    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675230    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675233    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675235    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675239    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675241    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675244    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675247    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675249    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675252    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675255    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675257    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675260    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675262    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675265    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675267    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675270    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675272    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675274    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:06:18.897808 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675277    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675279    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675283    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675285    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675289    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675292    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675295    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675298    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675301    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675303    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675306    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675309    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.675312    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676014    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676021    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676024    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676027    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676030    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676033    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676036    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:06:18.898612 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676039    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676043    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676047    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676051    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676054    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676057    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676059    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676062    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676064    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676067    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676070    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676072    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676075    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676077    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676080    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676082    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676085    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676088    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676090    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676094    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:06:18.899214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676096    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676099    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676105    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676108    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676111    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676114    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676116    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676119    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676121    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676124    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676127    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676134    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676137    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676139    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676142    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676145    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676148    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676150    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676153    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676156    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:06:18.899833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676158    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676161    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676163    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676166    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676169    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676171    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676174    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676177    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676179    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676182    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676184    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676187    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676189    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676192    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676195    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676198    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676200    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676203    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676207    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:06:18.900437 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676210    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676213    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676216    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676219    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676222    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676230    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676234    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676237    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676239    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676242    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676244    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676247    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676249    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676252    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676254    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676257    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676260    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676262    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676265    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:06:18.901046 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676267    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676270    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676273    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676275    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676277    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676280    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676282    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676286    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676291    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676294    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676296    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676299    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676301    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676304    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676663    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676668    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676670    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676673    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676676    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:06:18.901589 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676681    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676684    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676686    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676690    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676694    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676697    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676700    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676703    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676706    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676708    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676711    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676714    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676716    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676719    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676722    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676726    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676728    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676731    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676733    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:06:18.902253 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676736    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676738    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676742    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676745    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676747    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676750    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676752    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676755    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676758    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676761    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676763    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676766    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676768    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676773    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676776    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676778    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676780    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676783    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676786    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676788    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:06:18.902794 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676790    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676793    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676796    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676798    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676801    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676803    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676805    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676808    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676810    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676813    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676815    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676818    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676820    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676823    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676827    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676829    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676832    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676834    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676837    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676839    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:06:18.903374 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676842    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676844    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676847    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676849    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676852    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676856    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676859    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676861    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676864    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676866    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676869    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676871    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676874    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676876    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676879    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676882    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676885    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676887    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676890    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676892    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:06:18.903886 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676895    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676897    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676900    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676902    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676905    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676907    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676911    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676914    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676916    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676919    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676921    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676924    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676926    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676930    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.676933    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677013    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677020    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677027    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677032    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677036    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677039    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 18:06:18.904484 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677043    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677047    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677050    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677052    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677056    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677059    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677062    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677065    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677068    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677071    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677074    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677076    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677930    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677933    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677937    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677941    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677944    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677949    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677952    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677955    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677959    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677961    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677964    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677967    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 18:06:18.905049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677970    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677973    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677977    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677980    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677985    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.677987    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678003    2578 flags.go:64] FLAG: --enable-server="true"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678007    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678011    2578 flags.go:64] FLAG: --event-burst="100"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678014    2578 flags.go:64] FLAG: --event-qps="50"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678017    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678020    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678023    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678026    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678029    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678032    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678035    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678038    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678041    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678044    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678047    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678049    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678052    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678055    2578 flags.go:64] FLAG: --feature-gates=""
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678059    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 18:06:18.905687 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678064    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678067    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678070    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678073    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678076    2578 flags.go:64] FLAG: --help="false"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678078    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-132-161.ec2.internal"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678081    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678084    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678087    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678090    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678094    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678098    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678101    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678103    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678106    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678109    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678113    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678116    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678118    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678121    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678124    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678127    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678130    2578 flags.go:64] FLAG: --lock-file=""
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678132    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 18:06:18.906419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678135    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678138    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678143    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678146    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678149    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678151    2578 flags.go:64] FLAG: --logging-format="text"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678154    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678157    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678161    2578 flags.go:64] FLAG: --manifest-url=""
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678164    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678168    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678171    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678175    2578 flags.go:64] FLAG: --max-pods="110"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678178    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678180    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678183    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678186    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678189    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678192    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678196    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678203    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678206    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678209    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 18:06:18.907098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678212    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678216    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678221    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678223    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678226    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678229    2578 flags.go:64] FLAG: --port="10250"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678232    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678235    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dc19e8c813bb6af1"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678238    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678241    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678244    2578 flags.go:64] FLAG: --register-node="true"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678247    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678251    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678253    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678256    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678259    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678262    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678265    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678269    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678272    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678275    2578 flags.go:64] FLAG: --runonce="false"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678278    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678281    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678284    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678287    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 18:06:18.907987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678290    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678292    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678295    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678299    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678303    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678305    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678308    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678311    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678314    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678317    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678320    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678325    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678328    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678331    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678334    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678337    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678339    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678343    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678346    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678348    2578 flags.go:64] FLAG: --v="2"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678352    2578 flags.go:64] FLAG: --version="false"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678356    2578 flags.go:64] FLAG: --vmodule=""
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678360    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678363    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678468    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:06:18.908770 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678473    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678476    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678479    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678483    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678486    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678489    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678491    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678494    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678497    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678500    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678505    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678507    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678510    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678512    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678515    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678518    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678520    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678523    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678526    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678529    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:06:19.103713 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678532    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678534    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678537    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678540    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678544    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678547    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678550    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678554    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678556    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678559    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678562    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678565    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678568    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678571    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678573    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678576    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678579    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678581    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678584    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678587    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:06:19.104538 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678590    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678592    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678596    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678598    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678601    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678604    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678606    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678609    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678611    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678615    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678620    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678624    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678627    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678630    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678632    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678635    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678637    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678640    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678642    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:06:19.105149 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678645    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678647    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678650    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678653    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678655    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678661    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678664    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678667    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678670    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678673    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678675    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678679    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678683    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678686    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678689    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678693    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678696    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678698    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678701    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:06:19.105746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678704    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678707    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678709    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678712    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678714    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678718    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678720    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678722    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678725    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678727    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678730    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678732    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678735    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678737    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678740    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678742    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678745    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678747    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678751    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:06:19.106260 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.678753    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.678759    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.687393    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.687409    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.687422    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.687428    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687487    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687491    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687494    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687498    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687500    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687503    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687505    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687508    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687510    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:06:19.106793 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687513    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687516    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687518    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687521    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687524    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687526    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687529    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687532    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687535    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687537    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687539    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687542    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687545    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687547    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687550    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687552    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687555    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687558    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687560    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687563    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:06:19.107214 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687572    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687575    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687577    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687580    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687582    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687585    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687587    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687590    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687592    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687595    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687597    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687599    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687602    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687604    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687623    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687626    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687629    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687634    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687639    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687641    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:06:19.107776 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687644    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687646    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687649    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687651    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687654    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687657    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687659    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687662    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687666    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687669    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687672    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687674    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687677    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687686    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687689    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687691    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687694    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687697    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687699    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:06:19.108301 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687702    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687704    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687706    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687709    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687711    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687714    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687716    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687719    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687721    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687724    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687728    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687732    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687735    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687737    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687740    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687742    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687745    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687747    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687750    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687752    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:06:19.108833 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687755    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:06:19.109407 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687757    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:06:19.109407 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687760    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:06:19.109407 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687762    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:06:19.109407 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687766    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:06:19.109407 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687770    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:06:19.109407 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687773    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:06:19.109407 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687781    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:06:19.109407 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687784    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:06:19.109407 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687788    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:06:19.109407 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687791    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:06:19.109407 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.687796    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:06:19.109407 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687940    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:06:19.109407 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687944    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:06:19.109407 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687948    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687950    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687953    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687956    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687959    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687962    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687965    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687967    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687972    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687975    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687979    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687982    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687984    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.687987    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688003    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688006    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688009    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688011    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688014    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688017    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:06:19.109823 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688019    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688022    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688024    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688027    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688029    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688032    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688040    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688043    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688046    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688048    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688051    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688054    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688057    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688061    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688064    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688066    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688069    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688072    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688075    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688077    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:06:19.110358 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688081    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688084    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688088    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688090    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688093    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688095    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688098    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688100    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688103    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688105    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688108    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688110    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688113    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688115    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688118    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688121    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688123    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688126    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688128    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688137    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:06:19.110902 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688139    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688142    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688144    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688148    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688151    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688154    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688157    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688159    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688161    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688164    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688166    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688169    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688173    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688175    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688179    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688182    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688185    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688187    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688190    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:06:19.111443 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688192    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688195    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688197    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688200    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688202    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688205    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688207    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688210    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688212    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688215    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688218    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688220    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688223    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688231    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688233    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688237    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688240    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:06:19.111977 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:18.688243    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:06:19.112447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.688248    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:06:19.112447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.688377    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 18:06:19.112447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.690507    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 18:06:19.112447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.691760    2578 server.go:1019] "Starting client certificate rotation"
Jul 01 18:06:19.112447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.691860    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 18:06:19.112447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.691887    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 18:06:19.112447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.694146    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 18:06:19.112447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.699198    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 18:06:19.112447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.709525    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 01 18:06:19.112447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.711552    2578 log.go:25] "Validated CRI v1 image API"
Jul 01 18:06:19.112447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.712261    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 18:06:19.112447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.714653    2578 fs.go:135] Filesystem UUIDs: map[0c950852-0f12-453e-8b5c-c3a30cd9904a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d6d8f9e7-97d2-43b3-895d-a8da58e9f1e3:/dev/nvme0n1p4]
Jul 01 18:06:19.112854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.714672    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 18:06:19.112930 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.719401    2578 manager.go:216] Machine: {Timestamp:2026-07-01 18:06:18.718182973 +0000 UTC m=+0.285467943 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094291 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f76351ff1a725a21a3379e7ceffbd SystemUUID:ec2f7635-1ff1-a725-a21a-3379e7ceffbd BootID:6adc8d15-7dad-4756-a6e8-dd1ae2bcedcc Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c5:27:52:6f:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c5:27:52:6f:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:82:69:e4:20:22 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 18:06:19.112930 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.719494    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 18:06:19.112930 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.719581    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 18:06:19.112930 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.722222    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 18:06:19.112930 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.722242    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-161.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 18:06:19.112930 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.722809    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 18:06:19.112930 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.722819    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 18:06:19.112930 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.722826    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.723075    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.723406    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.724221    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.728172    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.728291    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.728322    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.728336    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.728350    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.728359    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.728367    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.728758    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.728781    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.730635    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.732111    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.732146    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.732323    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.732335    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.732341    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.732346    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.732354    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.732359    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.732678    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.732684    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.732694    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.732699    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.732708    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.732715    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.733317    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.733324    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.736116    2578 server.go:1268] "Started kubelet"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.736199    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.738327    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.739463    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.739628    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.741331    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.742896    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-161.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.743029    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.743055    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 18:06:19.113252 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.744655    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.744227    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-161.ec2.internal.18be3ca4463e6591  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-161.ec2.internal,UID:ip-10-0-132-161.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-161.ec2.internal,},FirstTimestamp:2026-07-01 18:06:18.736092561 +0000 UTC m=+0.303377530,LastTimestamp:2026-07-01 18:06:18.736092561 +0000 UTC m=+0.303377530,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-161.ec2.internal,}"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.747558    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.747562    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.747665    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-161.ec2.internal\" not found"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.748118    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.748132    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.748286    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.748296    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.748674    2578 factory.go:55] Registering systemd factory
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.748688    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.748465    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.748834    2578 factory.go:153] Registering CRI-O factory
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.748843    2578 factory.go:223] Registration of the crio container factory successfully
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.748894    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.748911    2578 factory.go:103] Registering Raw factory
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.748933    2578 manager.go:1196] Started watching for new ooms in manager
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.749306    2578 manager.go:349] Starting recovery of all containers
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.754904    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.754975    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.758634    2578 manager.go:354] Recovery completed
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.763190    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.765123    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.765151    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.765165    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.765601    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.765612    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.765630    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.765601    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.766338    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.766356    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:19.114669 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.766368    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.766944    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-161.ec2.internal.18be3ca447f99008  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-161.ec2.internal,UID:ip-10-0-132-161.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-161.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-161.ec2.internal,},FirstTimestamp:2026-07-01 18:06:18.76513588 +0000 UTC m=+0.332420851,LastTimestamp:2026-07-01 18:06:18.76513588 +0000 UTC m=+0.332420851,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-161.ec2.internal,}"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.767481    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jbj4n"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.767817    2578 policy_none.go:49] "None policy: Start"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.767830    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.767839    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.769848    2578 policy_none.go:47] "Start"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.776035    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jbj4n"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.777361    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-161.ec2.internal.18be3ca447f9e2a6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-161.ec2.internal,UID:ip-10-0-132-161.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-161.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-161.ec2.internal,},FirstTimestamp:2026-07-01 18:06:18.76515703 +0000 UTC m=+0.332442002,LastTimestamp:2026-07-01 18:06:18.76515703 +0000 UTC m=+0.332442002,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-161.ec2.internal,}"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.809511    2578 manager.go:337] "Starting Device Plugin manager"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.809536    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.809545    2578 server.go:87] "Starting device plugin registration server"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.809761    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.809773    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.809975    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.810062    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.810070    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.810583    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.810628    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-161.ec2.internal\" not found"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.813462    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.814827    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.814871    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.814889    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.814925    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.816726    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.910643    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.911457    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:19.248623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.911484    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.911496    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.911512    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.915593    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-161.ec2.internal"]
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.915639    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.916850    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.916872    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.916886    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.919045    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.919149    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.919173    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.919625    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.919636    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.919647    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.919657    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.919657    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.919687    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.920030    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.920051    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-161.ec2.internal\": node \"ip-10-0-132-161.ec2.internal\" not found"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.921783    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.921803    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.922432    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.922454    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:18.922468    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.942960    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-161.ec2.internal\" not found"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.963505    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-161.ec2.internal\" not found" node="ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:18.967776    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-161.ec2.internal\" not found" node="ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:19.043307    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-161.ec2.internal\" not found"
Jul 01 18:06:19.250653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.049578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/080a0830f4e17cb8b7f0213f1a606519-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal\" (UID: \"080a0830f4e17cb8b7f0213f1a606519\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.251447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.049600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/080a0830f4e17cb8b7f0213f1a606519-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal\" (UID: \"080a0830f4e17cb8b7f0213f1a606519\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.251447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.049622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0de41b0170b5099b66f5351f7020aac1-config\") pod \"kube-apiserver-proxy-ip-10-0-132-161.ec2.internal\" (UID: \"0de41b0170b5099b66f5351f7020aac1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.251447 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:19.143728    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-161.ec2.internal\" not found"
Jul 01 18:06:19.251447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.150057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/080a0830f4e17cb8b7f0213f1a606519-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal\" (UID: \"080a0830f4e17cb8b7f0213f1a606519\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.251447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.150083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0de41b0170b5099b66f5351f7020aac1-config\") pod \"kube-apiserver-proxy-ip-10-0-132-161.ec2.internal\" (UID: \"0de41b0170b5099b66f5351f7020aac1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.251447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.150099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/080a0830f4e17cb8b7f0213f1a606519-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal\" (UID: \"080a0830f4e17cb8b7f0213f1a606519\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.251447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.150134    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/080a0830f4e17cb8b7f0213f1a606519-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal\" (UID: \"080a0830f4e17cb8b7f0213f1a606519\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.251447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.150158    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/080a0830f4e17cb8b7f0213f1a606519-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal\" (UID: \"080a0830f4e17cb8b7f0213f1a606519\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.251447 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.150162    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0de41b0170b5099b66f5351f7020aac1-config\") pod \"kube-apiserver-proxy-ip-10-0-132-161.ec2.internal\" (UID: \"0de41b0170b5099b66f5351f7020aac1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.251447 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:19.244755    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-161.ec2.internal\" not found"
Jul 01 18:06:19.398595 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.267248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.398595 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.270394    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:19.398595 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:19.345473    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-161.ec2.internal\" not found"
Jul 01 18:06:19.583407 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:19.453144    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-161.ec2.internal\" not found"
Jul 01 18:06:19.583407 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:19.555115    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-161.ec2.internal\" not found"
Jul 01 18:06:19.759939 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.612536    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:06:19.759939 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.638589    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:06:19.759939 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:19.656173    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-161.ec2.internal\" not found"
Jul 01 18:06:19.759939 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.691375    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 18:06:19.759939 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.691481    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 18:06:19.759939 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.691520    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 18:06:19.759939 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.691529    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 18:06:19.759939 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:19.756248    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-161.ec2.internal\" not found"
Jul 01 18:06:19.897191 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.781579    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 18:01:19 +0000 UTC" deadline="2028-03-21 12:56:28.606545166 +0000 UTC"
Jul 01 18:06:19.897191 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.781604    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15090h50m8.824943749s"
Jul 01 18:06:19.897191 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:19.856953    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-161.ec2.internal\" not found"
Jul 01 18:06:19.897191 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.876481    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:19.897191 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.877537    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:19.897191 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.877570    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:19.897191 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.877580    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:20.068031 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:19.914276    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de41b0170b5099b66f5351f7020aac1.slice/crio-36114bfdc5c8da2b0bdaeacf08757515bdb078a5d24b2ec02105cdc3e57c26ea WatchSource:0}: Error finding container 36114bfdc5c8da2b0bdaeacf08757515bdb078a5d24b2ec02105cdc3e57c26ea: Status 404 returned error can't find the container with id 36114bfdc5c8da2b0bdaeacf08757515bdb078a5d24b2ec02105cdc3e57c26ea
Jul 01 18:06:20.068031 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:19.916395    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080a0830f4e17cb8b7f0213f1a606519.slice/crio-0dd3423fa41347138e4fc4a41aa7125a0900907b3f44692054fd97fca2a1c43a WatchSource:0}: Error finding container 0dd3423fa41347138e4fc4a41aa7125a0900907b3f44692054fd97fca2a1c43a: Status 404 returned error can't find the container with id 0dd3423fa41347138e4fc4a41aa7125a0900907b3f44692054fd97fca2a1c43a
Jul 01 18:06:20.068031 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:19.920769    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 18:06:20.068031 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:19.961065    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-161.ec2.internal\" not found"
Jul 01 18:06:20.068031 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.035028    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:06:20.068031 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.048057    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:20.068031 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.059427    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 18:06:20.068031 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.060546    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal"
Jul 01 18:06:20.072911 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.072738    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.730979    2578 apiserver.go:52] "Watching apiserver"
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.739547    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.740060    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-q89rm","kube-system/konnectivity-agent-x6rq6","kube-system/kube-apiserver-proxy-ip-10-0-132-161.ec2.internal","openshift-image-registry/node-ca-bbppr","openshift-multus/multus-additional-cni-plugins-dg6np","openshift-multus/network-metrics-daemon-2r8hd","openshift-network-operator/iptables-alerter-zgwh2","openshift-ovn-kubernetes/ovnkube-node-ldwxn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679","openshift-cluster-node-tuning-operator/tuned-67lbw","openshift-dns/node-resolver-hmcsv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal","openshift-multus/multus-rc8bn","openshift-network-diagnostics/network-check-target-t4vsn"]
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.742784    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zgwh2"
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.745182    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.747459    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:20.747530    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.748224    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.749301    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.749336    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.749462    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.749573    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.749737    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tts7v\""
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.749802    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.749908    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4tfcw\""
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.750006    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.750015    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.750191    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bbppr"
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.753683    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.753828    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.753970    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.754218    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bqn9z\""
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.754304    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.756485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.756535    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 18:06:20.788571 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:20.756565    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.756842    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.756865    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pnvfz\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.759054    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.761595    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.761649    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.761863    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vr9kh\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.761971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.762020    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.764687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x6rq6"
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.765977    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.766270    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.766476    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.766702    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.766876    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.767178    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.767365    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ntvm8\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.767835    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hkm2d\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.768021    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.768209    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.768554    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.764588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/583c6fb3-3031-43f8-bce9-aa90d151a40b-iptables-alerter-script\") pod \"iptables-alerter-zgwh2\" (UID: \"583c6fb3-3031-43f8-bce9-aa90d151a40b\") " pod="openshift-network-operator/iptables-alerter-zgwh2"
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.769814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6dkw\" (UniqueName: \"kubernetes.io/projected/583c6fb3-3031-43f8-bce9-aa90d151a40b-kube-api-access-m6dkw\") pod \"iptables-alerter-zgwh2\" (UID: \"583c6fb3-3031-43f8-bce9-aa90d151a40b\") " pod="openshift-network-operator/iptables-alerter-zgwh2"
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.769855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-run-netns\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.769893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-etc-kubernetes\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.791528 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.769929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d1e4c3c-8594-4a71-86fe-9e77d4f4153f-host\") pod \"node-ca-bbppr\" (UID: \"9d1e4c3c-8594-4a71-86fe-9e77d4f4153f\") " pod="openshift-image-registry/node-ca-bbppr"
Jul 01 18:06:20.792931 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.769957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-system-cni-dir\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.792931 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.769987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-cnibin\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.792931 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65be3311-bd63-439e-9980-ff65266cef62-cni-binary-copy\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.792931 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-var-lib-cni-bin\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.792931 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-var-lib-cni-multus\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.792931 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-hostroot\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.792931 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/65be3311-bd63-439e-9980-ff65266cef62-multus-daemon-config\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.792931 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-run-multus-certs\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.792931 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-multus-cni-dir\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.792931 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-os-release\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.792931 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-multus-socket-dir-parent\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.792931 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d1e4c3c-8594-4a71-86fe-9e77d4f4153f-serviceca\") pod \"node-ca-bbppr\" (UID: \"9d1e4c3c-8594-4a71-86fe-9e77d4f4153f\") " pod="openshift-image-registry/node-ca-bbppr"
Jul 01 18:06:20.792931 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df4c8\" (UniqueName: \"kubernetes.io/projected/9d1e4c3c-8594-4a71-86fe-9e77d4f4153f-kube-api-access-df4c8\") pod \"node-ca-bbppr\" (UID: \"9d1e4c3c-8594-4a71-86fe-9e77d4f4153f\") " pod="openshift-image-registry/node-ca-bbppr"
Jul 01 18:06:20.792931 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/583c6fb3-3031-43f8-bce9-aa90d151a40b-host-slash\") pod \"iptables-alerter-zgwh2\" (UID: \"583c6fb3-3031-43f8-bce9-aa90d151a40b\") " pod="openshift-network-operator/iptables-alerter-zgwh2"
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-run-k8s-cni-cncf-io\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-var-lib-kubelet\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-multus-conf-dir\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdj6x\" (UniqueName: \"kubernetes.io/projected/65be3311-bd63-439e-9980-ff65266cef62-kube-api-access-jdj6x\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.770876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvmqz\" (UniqueName: \"kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz\") pod \"network-check-target-t4vsn\" (UID: \"9ceed3fb-b91c-43c2-8813-7991a3c27659\") " pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.771450    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.774514    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.774605    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s5d6x\""
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.774762    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.776318    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hmcsv"
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.778665    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-49hlv\""
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.778945    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.778965    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:20.779053    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.779096    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 18:06:20.794788 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.791388    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t2rfn"
Jul 01 18:06:20.801101 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.799350    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t2rfn"
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.819938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal" event={"ID":"080a0830f4e17cb8b7f0213f1a606519","Type":"ContainerStarted","Data":"0dd3423fa41347138e4fc4a41aa7125a0900907b3f44692054fd97fca2a1c43a"}
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.821139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-161.ec2.internal" event={"ID":"0de41b0170b5099b66f5351f7020aac1","Type":"ContainerStarted","Data":"36114bfdc5c8da2b0bdaeacf08757515bdb078a5d24b2ec02105cdc3e57c26ea"}
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.849849    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-multus-cni-dir\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d1e4c3c-8594-4a71-86fe-9e77d4f4153f-serviceca\") pod \"node-ca-bbppr\" (UID: \"9d1e4c3c-8594-4a71-86fe-9e77d4f4153f\") " pod="openshift-image-registry/node-ca-bbppr"
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4ce225ff-82c4-49f3-9dd0-b8df3d1b196d-hosts-file\") pod \"node-resolver-hmcsv\" (UID: \"4ce225ff-82c4-49f3-9dd0-b8df3d1b196d\") " pod="openshift-dns/node-resolver-hmcsv"
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5f3eafe-b7a2-47e9-954a-028746276935-dbus\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-env-overrides\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lck89\" (UniqueName: \"kubernetes.io/projected/a7450108-384d-45a2-b88d-ac61a546bab4-kube-api-access-lck89\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbs8t\" (UniqueName: \"kubernetes.io/projected/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-kube-api-access-jbs8t\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7223ef7-a5f6-4690-be25-9bed08246516-konnectivity-ca\") pod \"konnectivity-agent-x6rq6\" (UID: \"f7223ef7-a5f6-4690-be25-9bed08246516\") " pod="kube-system/konnectivity-agent-x6rq6"
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871361    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-multus-cni-dir\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-kubelet\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.890560 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-run-netns\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.891475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-run-systemd\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.891475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-registration-dir\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:20.891475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/298d79e6-cc04-47e5-98a5-36e2284c5174-cnibin\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:20.891475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-var-lib-cni-multus\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.891475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/65be3311-bd63-439e-9980-ff65266cef62-multus-daemon-config\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.891475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871574    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-var-lib-cni-multus\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.891475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gddtf\" (UniqueName: \"kubernetes.io/projected/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-kube-api-access-gddtf\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.891475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-sys-fs\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:20.891475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/298d79e6-cc04-47e5-98a5-36e2284c5174-cni-binary-copy\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:20.891475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-run-ovn\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.891475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d1e4c3c-8594-4a71-86fe-9e77d4f4153f-host\") pod \"node-ca-bbppr\" (UID: \"9d1e4c3c-8594-4a71-86fe-9e77d4f4153f\") " pod="openshift-image-registry/node-ca-bbppr"
Jul 01 18:06:20.891475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871760    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d1e4c3c-8594-4a71-86fe-9e77d4f4153f-host\") pod \"node-ca-bbppr\" (UID: \"9d1e4c3c-8594-4a71-86fe-9e77d4f4153f\") " pod="openshift-image-registry/node-ca-bbppr"
Jul 01 18:06:20.891475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/583c6fb3-3031-43f8-bce9-aa90d151a40b-host-slash\") pod \"iptables-alerter-zgwh2\" (UID: \"583c6fb3-3031-43f8-bce9-aa90d151a40b\") " pod="openshift-network-operator/iptables-alerter-zgwh2"
Jul 01 18:06:20.891475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-run-k8s-cni-cncf-io\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.893749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-var-lib-kubelet\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.893749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871879    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/583c6fb3-3031-43f8-bce9-aa90d151a40b-host-slash\") pod \"iptables-alerter-zgwh2\" (UID: \"583c6fb3-3031-43f8-bce9-aa90d151a40b\") " pod="openshift-network-operator/iptables-alerter-zgwh2"
Jul 01 18:06:20.893749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-var-lib-openvswitch\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.893749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871834    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d1e4c3c-8594-4a71-86fe-9e77d4f4153f-serviceca\") pod \"node-ca-bbppr\" (UID: \"9d1e4c3c-8594-4a71-86fe-9e77d4f4153f\") " pod="openshift-image-registry/node-ca-bbppr"
Jul 01 18:06:20.893749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-lib-modules\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.893749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/583c6fb3-3031-43f8-bce9-aa90d151a40b-iptables-alerter-script\") pod \"iptables-alerter-zgwh2\" (UID: \"583c6fb3-3031-43f8-bce9-aa90d151a40b\") " pod="openshift-network-operator/iptables-alerter-zgwh2"
Jul 01 18:06:20.893749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871943    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-run-k8s-cni-cncf-io\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.893749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:20.893749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:20.893749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-socket-dir\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:20.893749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.871968    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-var-lib-kubelet\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.893749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-run-openvswitch\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.893749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-cnibin\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.893749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ce225ff-82c4-49f3-9dd0-b8df3d1b196d-tmp-dir\") pod \"node-resolver-hmcsv\" (UID: \"4ce225ff-82c4-49f3-9dd0-b8df3d1b196d\") " pod="openshift-dns/node-resolver-hmcsv"
Jul 01 18:06:20.894689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-device-dir\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:20.894689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872135    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/65be3311-bd63-439e-9980-ff65266cef62-multus-daemon-config\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.894689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-node-log\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.894689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.894689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-kubernetes\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.894689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872214    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-cnibin\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.894689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-os-release\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.894689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-multus-socket-dir-parent\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.894689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgl58\" (UniqueName: \"kubernetes.io/projected/298d79e6-cc04-47e5-98a5-36e2284c5174-kube-api-access-tgl58\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:20.894689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-multus-conf-dir\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.894689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdj6x\" (UniqueName: \"kubernetes.io/projected/65be3311-bd63-439e-9980-ff65266cef62-kube-api-access-jdj6x\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.894689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872319    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-os-release\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.894689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872325    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-multus-socket-dir-parent\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.894689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqz\" (UniqueName: \"kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz\") pod \"network-check-target-t4vsn\" (UID: \"9ceed3fb-b91c-43c2-8813-7991a3c27659\") " pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:20.896992 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/298d79e6-cc04-47e5-98a5-36e2284c5174-system-cni-dir\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:20.896992 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/298d79e6-cc04-47e5-98a5-36e2284c5174-os-release\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:20.896992 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-log-socket\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.896992 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-run\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.896992 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-var-lib-kubelet\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.896992 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872492    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/583c6fb3-3031-43f8-bce9-aa90d151a40b-iptables-alerter-script\") pod \"iptables-alerter-zgwh2\" (UID: \"583c6fb3-3031-43f8-bce9-aa90d151a40b\") " pod="openshift-network-operator/iptables-alerter-zgwh2"
Jul 01 18:06:20.896992 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6dkw\" (UniqueName: \"kubernetes.io/projected/583c6fb3-3031-43f8-bce9-aa90d151a40b-kube-api-access-m6dkw\") pod \"iptables-alerter-zgwh2\" (UID: \"583c6fb3-3031-43f8-bce9-aa90d151a40b\") " pod="openshift-network-operator/iptables-alerter-zgwh2"
Jul 01 18:06:20.896992 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-run-netns\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.896992 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872543    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-multus-conf-dir\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.896992 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5f3eafe-b7a2-47e9-954a-028746276935-kubelet-config\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:20.896992 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65be3311-bd63-439e-9980-ff65266cef62-cni-binary-copy\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.896992 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872587    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-run-netns\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.896992 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-var-lib-cni-bin\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.896992 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-run-multus-certs\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.898599 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:20.898599 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-cni-netd\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.898599 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872677    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-var-lib-cni-bin\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.898599 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-ovn-node-metrics-cert\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.898599 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872716    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-host-run-multus-certs\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.898599 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df4c8\" (UniqueName: \"kubernetes.io/projected/9d1e4c3c-8594-4a71-86fe-9e77d4f4153f-kube-api-access-df4c8\") pod \"node-ca-bbppr\" (UID: \"9d1e4c3c-8594-4a71-86fe-9e77d4f4153f\") " pod="openshift-image-registry/node-ca-bbppr"
Jul 01 18:06:20.898599 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thmkd\" (UniqueName: \"kubernetes.io/projected/8936baa9-7b61-4534-b564-a7bb12bf3535-kube-api-access-thmkd\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:20.898599 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-ovnkube-config\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.898599 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-ovnkube-script-lib\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.898599 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-sysctl-d\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.898599 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-sysctl-conf\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.898599 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-systemd\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.898599 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-sys\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.898599 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/298d79e6-cc04-47e5-98a5-36e2284c5174-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:20.901854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7223ef7-a5f6-4690-be25-9bed08246516-agent-certs\") pod \"konnectivity-agent-x6rq6\" (UID: \"f7223ef7-a5f6-4690-be25-9bed08246516\") " pod="kube-system/konnectivity-agent-x6rq6"
Jul 01 18:06:20.901854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/298d79e6-cc04-47e5-98a5-36e2284c5174-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:20.901854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-etc-openvswitch\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.901854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.872992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-sysconfig\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.901854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7450108-384d-45a2-b88d-ac61a546bab4-etc-tuned\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.901854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873045    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65be3311-bd63-439e-9980-ff65266cef62-cni-binary-copy\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.901854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-etc-kubernetes\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.901854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873106    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-etc-kubernetes\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.901854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bf7k\" (UniqueName: \"kubernetes.io/projected/4ce225ff-82c4-49f3-9dd0-b8df3d1b196d-kube-api-access-4bf7k\") pod \"node-resolver-hmcsv\" (UID: \"4ce225ff-82c4-49f3-9dd0-b8df3d1b196d\") " pod="openshift-dns/node-resolver-hmcsv"
Jul 01 18:06:20.901854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/298d79e6-cc04-47e5-98a5-36e2284c5174-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:20.901854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-slash\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.901854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.901854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-system-cni-dir\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.901854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-hostroot\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-systemd-units\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873322    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-hostroot\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873325    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65be3311-bd63-439e-9980-ff65266cef62-system-cni-dir\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873336    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-cni-bin\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-modprobe-d\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-host\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.873407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7450108-384d-45a2-b88d-ac61a546bab4-tmp\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.883606    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:20.883639    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:20.883660    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:20.883673    2578 projected.go:196] Error preparing data for projected volume kube-api-access-wvmqz for pod openshift-network-diagnostics/network-check-target-t4vsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:20.883735    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz podName:9ceed3fb-b91c-43c2-8813-7991a3c27659 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:21.383705745 +0000 UTC m=+2.950990708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wvmqz" (UniqueName: "kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz") pod "network-check-target-t4vsn" (UID: "9ceed3fb-b91c-43c2-8813-7991a3c27659") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.888093    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6dkw\" (UniqueName: \"kubernetes.io/projected/583c6fb3-3031-43f8-bce9-aa90d151a40b-kube-api-access-m6dkw\") pod \"iptables-alerter-zgwh2\" (UID: \"583c6fb3-3031-43f8-bce9-aa90d151a40b\") " pod="openshift-network-operator/iptables-alerter-zgwh2"
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.888117    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df4c8\" (UniqueName: \"kubernetes.io/projected/9d1e4c3c-8594-4a71-86fe-9e77d4f4153f-kube-api-access-df4c8\") pod \"node-ca-bbppr\" (UID: \"9d1e4c3c-8594-4a71-86fe-9e77d4f4153f\") " pod="openshift-image-registry/node-ca-bbppr"
Jul 01 18:06:20.902794 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.888117    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdj6x\" (UniqueName: \"kubernetes.io/projected/65be3311-bd63-439e-9980-ff65266cef62-kube-api-access-jdj6x\") pod \"multus-rc8bn\" (UID: \"65be3311-bd63-439e-9980-ff65266cef62\") " pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:20.996412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-var-lib-openvswitch\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.996412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-lib-modules\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.996412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:20.996412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:20.996412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-socket-dir\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:20.996412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-run-openvswitch\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.996412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ce225ff-82c4-49f3-9dd0-b8df3d1b196d-tmp-dir\") pod \"node-resolver-hmcsv\" (UID: \"4ce225ff-82c4-49f3-9dd0-b8df3d1b196d\") " pod="openshift-dns/node-resolver-hmcsv"
Jul 01 18:06:20.996412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-device-dir\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:20.996412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-node-log\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.996412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.996412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-kubernetes\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.996412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgl58\" (UniqueName: \"kubernetes.io/projected/298d79e6-cc04-47e5-98a5-36e2284c5174-kube-api-access-tgl58\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:20.996412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/298d79e6-cc04-47e5-98a5-36e2284c5174-system-cni-dir\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:20.996412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/298d79e6-cc04-47e5-98a5-36e2284c5174-os-release\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:20.998816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-log-socket\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.998816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-run\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.998816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-var-lib-kubelet\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.998816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5f3eafe-b7a2-47e9-954a-028746276935-kubelet-config\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:20.998816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:20.998816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-cni-netd\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.998816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-ovn-node-metrics-cert\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.998816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thmkd\" (UniqueName: \"kubernetes.io/projected/8936baa9-7b61-4534-b564-a7bb12bf3535-kube-api-access-thmkd\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:20.998816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-ovnkube-config\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.998816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-ovnkube-script-lib\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.998816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974861    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-sysctl-d\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.998816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974890    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-sysctl-conf\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.998816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-systemd\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.998816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-sys\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.999757 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/298d79e6-cc04-47e5-98a5-36e2284c5174-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:20.999757 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.974993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7223ef7-a5f6-4690-be25-9bed08246516-agent-certs\") pod \"konnectivity-agent-x6rq6\" (UID: \"f7223ef7-a5f6-4690-be25-9bed08246516\") " pod="kube-system/konnectivity-agent-x6rq6"
Jul 01 18:06:20.999757 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.975018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/298d79e6-cc04-47e5-98a5-36e2284c5174-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:20.999757 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.975044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-etc-openvswitch\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.999757 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.975153    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-sysconfig\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.999757 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.975200    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-var-lib-openvswitch\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.999757 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.975304    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-lib-modules\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.999757 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:20.975385    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:20.999757 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:20.975445    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs podName:8936baa9-7b61-4534-b564-a7bb12bf3535 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:21.475420877 +0000 UTC m=+3.042705840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs") pod "network-metrics-daemon-2r8hd" (UID: "8936baa9-7b61-4534-b564-a7bb12bf3535") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:20.999757 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.975650    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:20.999757 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.975759    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-socket-dir\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:20.999757 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.975802    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-run-openvswitch\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:20.999757 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.975952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-sysconfig\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:20.999757 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.975992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7450108-384d-45a2-b88d-ac61a546bab4-etc-tuned\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bf7k\" (UniqueName: \"kubernetes.io/projected/4ce225ff-82c4-49f3-9dd0-b8df3d1b196d-kube-api-access-4bf7k\") pod \"node-resolver-hmcsv\" (UID: \"4ce225ff-82c4-49f3-9dd0-b8df3d1b196d\") " pod="openshift-dns/node-resolver-hmcsv"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976060    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ce225ff-82c4-49f3-9dd0-b8df3d1b196d-tmp-dir\") pod \"node-resolver-hmcsv\" (UID: \"4ce225ff-82c4-49f3-9dd0-b8df3d1b196d\") " pod="openshift-dns/node-resolver-hmcsv"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/298d79e6-cc04-47e5-98a5-36e2284c5174-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-slash\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976165    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-device-dir\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-systemd-units\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976207    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-node-log\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-cni-bin\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976248    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-modprobe-d\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-host\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976296    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-kubernetes\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7450108-384d-45a2-b88d-ac61a546bab4-tmp\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.082529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4ce225ff-82c4-49f3-9dd0-b8df3d1b196d-hosts-file\") pod \"node-resolver-hmcsv\" (UID: \"4ce225ff-82c4-49f3-9dd0-b8df3d1b196d\") " pod="openshift-dns/node-resolver-hmcsv"
Jul 01 18:06:21.083640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5f3eafe-b7a2-47e9-954a-028746276935-dbus\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:21.083640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-env-overrides\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.083640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lck89\" (UniqueName: \"kubernetes.io/projected/a7450108-384d-45a2-b88d-ac61a546bab4-kube-api-access-lck89\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.083640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:21.083640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbs8t\" (UniqueName: \"kubernetes.io/projected/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-kube-api-access-jbs8t\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:21.083640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7223ef7-a5f6-4690-be25-9bed08246516-konnectivity-ca\") pod \"konnectivity-agent-x6rq6\" (UID: \"f7223ef7-a5f6-4690-be25-9bed08246516\") " pod="kube-system/konnectivity-agent-x6rq6"
Jul 01 18:06:21.083640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976524    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/298d79e6-cc04-47e5-98a5-36e2284c5174-system-cni-dir\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:21.083640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-kubelet\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.083640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-run-netns\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.083640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-run-systemd\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.083640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-registration-dir\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:21.083640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/298d79e6-cc04-47e5-98a5-36e2284c5174-cnibin\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:21.083640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gddtf\" (UniqueName: \"kubernetes.io/projected/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-kube-api-access-gddtf\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.083640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976671    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/298d79e6-cc04-47e5-98a5-36e2284c5174-os-release\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-sys-fs\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976721    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-log-socket\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/298d79e6-cc04-47e5-98a5-36e2284c5174-cni-binary-copy\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976766    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-run\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-run-ovn\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976805    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-run-ovn\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.976852    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-var-lib-kubelet\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.977182    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/298d79e6-cc04-47e5-98a5-36e2284c5174-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.977233    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-slash\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.977274    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.977314    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-systemd-units\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.977351    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-cni-bin\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.977440    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-modprobe-d\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.977488    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-host\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.084333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.978060    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-sysctl-d\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.085677 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.978143    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5f3eafe-b7a2-47e9-954a-028746276935-kubelet-config\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:21.085677 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:20.978530    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:21.085677 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.978547    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-cni-netd\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.085677 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:20.978588    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret podName:c5f3eafe-b7a2-47e9-954a-028746276935 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:21.478572704 +0000 UTC m=+3.045857662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret") pod "global-pull-secret-syncer-q89rm" (UID: "c5f3eafe-b7a2-47e9-954a-028746276935") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:21.085677 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.978629    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-kubelet\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.085677 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.978675    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4ce225ff-82c4-49f3-9dd0-b8df3d1b196d-hosts-file\") pod \"node-resolver-hmcsv\" (UID: \"4ce225ff-82c4-49f3-9dd0-b8df3d1b196d\") " pod="openshift-dns/node-resolver-hmcsv"
Jul 01 18:06:21.085677 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.978719    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5f3eafe-b7a2-47e9-954a-028746276935-dbus\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:21.085677 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.978984    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-ovnkube-config\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.085677 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.979047    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-env-overrides\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.085677 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.979088    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/298d79e6-cc04-47e5-98a5-36e2284c5174-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:21.085677 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.979191    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:21.085677 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.979497    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-sysctl-conf\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.085677 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.979520    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7223ef7-a5f6-4690-be25-9bed08246516-konnectivity-ca\") pod \"konnectivity-agent-x6rq6\" (UID: \"f7223ef7-a5f6-4690-be25-9bed08246516\") " pod="kube-system/konnectivity-agent-x6rq6"
Jul 01 18:06:21.085677 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.979526    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-ovnkube-script-lib\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.979557    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-etc-systemd\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.979609    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7450108-384d-45a2-b88d-ac61a546bab4-sys\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.979823    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-etc-openvswitch\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.979873    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/298d79e6-cc04-47e5-98a5-36e2284c5174-cnibin\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.979891    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-sys-fs\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.979916    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-host-run-netns\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.979944    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-run-systemd\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.979961    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-registration-dir\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.980407    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/298d79e6-cc04-47e5-98a5-36e2284c5174-cni-binary-copy\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.980555    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/298d79e6-cc04-47e5-98a5-36e2284c5174-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.983039    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7450108-384d-45a2-b88d-ac61a546bab4-tmp\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.983218    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7450108-384d-45a2-b88d-ac61a546bab4-etc-tuned\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.984685    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7223ef7-a5f6-4690-be25-9bed08246516-agent-certs\") pod \"konnectivity-agent-x6rq6\" (UID: \"f7223ef7-a5f6-4690-be25-9bed08246516\") " pod="kube-system/konnectivity-agent-x6rq6"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.985632    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-ovn-node-metrics-cert\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.086276 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.986155    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thmkd\" (UniqueName: \"kubernetes.io/projected/8936baa9-7b61-4534-b564-a7bb12bf3535-kube-api-access-thmkd\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:21.087097 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.987808    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgl58\" (UniqueName: \"kubernetes.io/projected/298d79e6-cc04-47e5-98a5-36e2284c5174-kube-api-access-tgl58\") pod \"multus-additional-cni-plugins-dg6np\" (UID: \"298d79e6-cc04-47e5-98a5-36e2284c5174\") " pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:21.087097 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.988449    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbs8t\" (UniqueName: \"kubernetes.io/projected/dcc9a1bf-40a4-4775-9eef-fcc8e52257ce-kube-api-access-jbs8t\") pod \"aws-ebs-csi-driver-node-br679\" (UID: \"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:21.087097 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.989124    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bf7k\" (UniqueName: \"kubernetes.io/projected/4ce225ff-82c4-49f3-9dd0-b8df3d1b196d-kube-api-access-4bf7k\") pod \"node-resolver-hmcsv\" (UID: \"4ce225ff-82c4-49f3-9dd0-b8df3d1b196d\") " pod="openshift-dns/node-resolver-hmcsv"
Jul 01 18:06:21.087097 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.989221    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gddtf\" (UniqueName: \"kubernetes.io/projected/11a3da3b-b7dc-4aed-b261-597c2fc14fdb-kube-api-access-gddtf\") pod \"ovnkube-node-ldwxn\" (UID: \"11a3da3b-b7dc-4aed-b261-597c2fc14fdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.087097 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:20.989668    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lck89\" (UniqueName: \"kubernetes.io/projected/a7450108-384d-45a2-b88d-ac61a546bab4-kube-api-access-lck89\") pod \"tuned-67lbw\" (UID: \"a7450108-384d-45a2-b88d-ac61a546bab4\") " pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.131238 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.131209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zgwh2"
Jul 01 18:06:21.133910 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.133892    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:06:21.142171 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.142155    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rc8bn"
Jul 01 18:06:21.149084 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.149050    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bbppr"
Jul 01 18:06:21.153954 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.153934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dg6np"
Jul 01 18:06:21.158136 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:21.158110    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1e4c3c_8594_4a71_86fe_9e77d4f4153f.slice/crio-aa5164ff3d48aac24302199a80b8f528075631d2d8a91b959201b0e97c03de16 WatchSource:0}: Error finding container aa5164ff3d48aac24302199a80b8f528075631d2d8a91b959201b0e97c03de16: Status 404 returned error can't find the container with id aa5164ff3d48aac24302199a80b8f528075631d2d8a91b959201b0e97c03de16
Jul 01 18:06:21.161785 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.161664    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679"
Jul 01 18:06:21.166236 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:21.166208    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298d79e6_cc04_47e5_98a5_36e2284c5174.slice/crio-247bb3ce5f733491248536b8502f9ad8d40963242dfcf5a61bec01b90e58b842 WatchSource:0}: Error finding container 247bb3ce5f733491248536b8502f9ad8d40963242dfcf5a61bec01b90e58b842: Status 404 returned error can't find the container with id 247bb3ce5f733491248536b8502f9ad8d40963242dfcf5a61bec01b90e58b842
Jul 01 18:06:21.169388 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.169368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:21.171720 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:21.171682    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc9a1bf_40a4_4775_9eef_fcc8e52257ce.slice/crio-55ad258680720bf5889c3cb5aeac386d2bbe3f08f1a55f4ebc4f051159a4c565 WatchSource:0}: Error finding container 55ad258680720bf5889c3cb5aeac386d2bbe3f08f1a55f4ebc4f051159a4c565: Status 404 returned error can't find the container with id 55ad258680720bf5889c3cb5aeac386d2bbe3f08f1a55f4ebc4f051159a4c565
Jul 01 18:06:21.178631 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.178611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x6rq6"
Jul 01 18:06:21.180734 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:21.180709    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a3da3b_b7dc_4aed_b261_597c2fc14fdb.slice/crio-79dd9bc31d47828e1b70e078ed2a6446db959380c47c2e0d10f0bd7f43d24b96 WatchSource:0}: Error finding container 79dd9bc31d47828e1b70e078ed2a6446db959380c47c2e0d10f0bd7f43d24b96: Status 404 returned error can't find the container with id 79dd9bc31d47828e1b70e078ed2a6446db959380c47c2e0d10f0bd7f43d24b96
Jul 01 18:06:21.186218 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.186197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-67lbw"
Jul 01 18:06:21.189658 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:21.189635    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7223ef7_a5f6_4690_be25_9bed08246516.slice/crio-76ac2eadce548c9d0bac7c23ec0a6db187b38d1a1cd821adca58de781b00f6c8 WatchSource:0}: Error finding container 76ac2eadce548c9d0bac7c23ec0a6db187b38d1a1cd821adca58de781b00f6c8: Status 404 returned error can't find the container with id 76ac2eadce548c9d0bac7c23ec0a6db187b38d1a1cd821adca58de781b00f6c8
Jul 01 18:06:21.196379 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:21.196354    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7450108_384d_45a2_b88d_ac61a546bab4.slice/crio-1f4612420c6d12c9cdca9bc44f94a8891b9a790adbdab1d24fceda82ba6353e0 WatchSource:0}: Error finding container 1f4612420c6d12c9cdca9bc44f94a8891b9a790adbdab1d24fceda82ba6353e0: Status 404 returned error can't find the container with id 1f4612420c6d12c9cdca9bc44f94a8891b9a790adbdab1d24fceda82ba6353e0
Jul 01 18:06:21.202056 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.202035    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hmcsv"
Jul 01 18:06:21.480328 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.480244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:21.480328 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.480289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqz\" (UniqueName: \"kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz\") pod \"network-check-target-t4vsn\" (UID: \"9ceed3fb-b91c-43c2-8813-7991a3c27659\") " pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:21.480328 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.480323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:21.480600 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:21.480433    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:21.480600 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:21.480448    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:06:21.480600 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:21.480465    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:06:21.480600 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:21.480467    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:21.480600 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:21.480500    2578 projected.go:196] Error preparing data for projected volume kube-api-access-wvmqz for pod openshift-network-diagnostics/network-check-target-t4vsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:21.480600 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:21.480513    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs podName:8936baa9-7b61-4534-b564-a7bb12bf3535 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:22.480487623 +0000 UTC m=+4.047772601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs") pod "network-metrics-daemon-2r8hd" (UID: "8936baa9-7b61-4534-b564-a7bb12bf3535") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:21.480600 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:21.480535    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret podName:c5f3eafe-b7a2-47e9-954a-028746276935 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:22.480522823 +0000 UTC m=+4.047807782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret") pod "global-pull-secret-syncer-q89rm" (UID: "c5f3eafe-b7a2-47e9-954a-028746276935") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:21.480600 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:21.480552    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz podName:9ceed3fb-b91c-43c2-8813-7991a3c27659 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:22.48054115 +0000 UTC m=+4.047826132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvmqz" (UniqueName: "kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz") pod "network-check-target-t4vsn" (UID: "9ceed3fb-b91c-43c2-8813-7991a3c27659") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:21.802801 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.802629    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 18:01:21 +0000 UTC" deadline="2028-03-31 00:10:57.988021272 +0000 UTC"
Jul 01 18:06:21.802801 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.802707    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15318h4m36.18531952s"
Jul 01 18:06:21.824764 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.824721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-67lbw" event={"ID":"a7450108-384d-45a2-b88d-ac61a546bab4","Type":"ContainerStarted","Data":"1f4612420c6d12c9cdca9bc44f94a8891b9a790adbdab1d24fceda82ba6353e0"}
Jul 01 18:06:21.826378 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.826312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x6rq6" event={"ID":"f7223ef7-a5f6-4690-be25-9bed08246516","Type":"ContainerStarted","Data":"76ac2eadce548c9d0bac7c23ec0a6db187b38d1a1cd821adca58de781b00f6c8"}
Jul 01 18:06:21.827905 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.827875    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn" event={"ID":"11a3da3b-b7dc-4aed-b261-597c2fc14fdb","Type":"ContainerStarted","Data":"79dd9bc31d47828e1b70e078ed2a6446db959380c47c2e0d10f0bd7f43d24b96"}
Jul 01 18:06:21.829204 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.829152    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg6np" event={"ID":"298d79e6-cc04-47e5-98a5-36e2284c5174","Type":"ContainerStarted","Data":"247bb3ce5f733491248536b8502f9ad8d40963242dfcf5a61bec01b90e58b842"}
Jul 01 18:06:21.830807 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.830760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bbppr" event={"ID":"9d1e4c3c-8594-4a71-86fe-9e77d4f4153f","Type":"ContainerStarted","Data":"aa5164ff3d48aac24302199a80b8f528075631d2d8a91b959201b0e97c03de16"}
Jul 01 18:06:21.832091 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.832064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rc8bn" event={"ID":"65be3311-bd63-439e-9980-ff65266cef62","Type":"ContainerStarted","Data":"81459e16fcf7435f57c9cd73289e8ef441e282579f54e4cf284ef5cbe77d7fa6"}
Jul 01 18:06:21.833591 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.833569    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679" event={"ID":"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce","Type":"ContainerStarted","Data":"55ad258680720bf5889c3cb5aeac386d2bbe3f08f1a55f4ebc4f051159a4c565"}
Jul 01 18:06:21.834987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.834959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zgwh2" event={"ID":"583c6fb3-3031-43f8-bce9-aa90d151a40b","Type":"ContainerStarted","Data":"b38e46aa5ce37ceef27091901843a3728fff2f1b043d8887962cda7f24d67c33"}
Jul 01 18:06:21.982018 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:21.981979    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:06:22.488615 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:22.487944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:22.488615 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:22.487996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqz\" (UniqueName: \"kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz\") pod \"network-check-target-t4vsn\" (UID: \"9ceed3fb-b91c-43c2-8813-7991a3c27659\") " pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:22.488615 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:22.488054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:22.488615 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:22.488188    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:22.488615 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:22.488250    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret podName:c5f3eafe-b7a2-47e9-954a-028746276935 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:24.488229762 +0000 UTC m=+6.055514725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret") pod "global-pull-secret-syncer-q89rm" (UID: "c5f3eafe-b7a2-47e9-954a-028746276935") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:22.488615 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:22.488414    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:06:22.488615 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:22.488432    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:06:22.488615 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:22.488444    2578 projected.go:196] Error preparing data for projected volume kube-api-access-wvmqz for pod openshift-network-diagnostics/network-check-target-t4vsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:22.488615 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:22.488487    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz podName:9ceed3fb-b91c-43c2-8813-7991a3c27659 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:24.488473086 +0000 UTC m=+6.055758059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvmqz" (UniqueName: "kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz") pod "network-check-target-t4vsn" (UID: "9ceed3fb-b91c-43c2-8813-7991a3c27659") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:22.488615 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:22.488553    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:22.488615 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:22.488590    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs podName:8936baa9-7b61-4534-b564-a7bb12bf3535 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:24.488577503 +0000 UTC m=+6.055862477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs") pod "network-metrics-daemon-2r8hd" (UID: "8936baa9-7b61-4534-b564-a7bb12bf3535") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:22.698646 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:22.698249    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:06:22.803142 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:22.802993    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 18:01:21 +0000 UTC" deadline="2028-01-18 15:01:32.254597175 +0000 UTC"
Jul 01 18:06:22.803142 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:22.803035    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13580h55m9.451570098s"
Jul 01 18:06:22.816242 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:22.815476    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:22.816242 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:22.815632    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:22.816242 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:22.816061    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:22.816242 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:22.816202    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:22.816242 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:22.816219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:22.816621 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:22.816349    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:22.848812 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:22.848774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-161.ec2.internal" event={"ID":"0de41b0170b5099b66f5351f7020aac1","Type":"ContainerStarted","Data":"839edec410e7fe2af32048c5c10f2d27a0bdc9506506c8d791cbdab2475dfa09"}
Jul 01 18:06:22.853254 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:22.853197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hmcsv" event={"ID":"4ce225ff-82c4-49f3-9dd0-b8df3d1b196d","Type":"ContainerStarted","Data":"83efd97e7db4774203b4c7cedc2646fce83401070183d830644fa70a9f917dfb"}
Jul 01 18:06:23.860221 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:23.860136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal" event={"ID":"080a0830f4e17cb8b7f0213f1a606519","Type":"ContainerStarted","Data":"7c07945c9f4ca08f08b7313170bc9d91b56876f47ada82ac09ddb7100369349a"}
Jul 01 18:06:23.884664 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:23.882086    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-161.ec2.internal" podStartSLOduration=3.8820670550000003 podStartE2EDuration="3.882067055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:22.865094763 +0000 UTC m=+4.432379746" watchObservedRunningTime="2026-07-01 18:06:23.882067055 +0000 UTC m=+5.449352038"
Jul 01 18:06:24.507186 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:24.507150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:24.507374 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:24.507212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqz\" (UniqueName: \"kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz\") pod \"network-check-target-t4vsn\" (UID: \"9ceed3fb-b91c-43c2-8813-7991a3c27659\") " pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:24.507374 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:24.507248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:24.507374 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:24.507363    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:24.507551 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:24.507423    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs podName:8936baa9-7b61-4534-b564-a7bb12bf3535 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:28.507408368 +0000 UTC m=+10.074693327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs") pod "network-metrics-daemon-2r8hd" (UID: "8936baa9-7b61-4534-b564-a7bb12bf3535") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:24.507551 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:24.507448    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:06:24.507551 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:24.507464    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:06:24.507551 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:24.507476    2578 projected.go:196] Error preparing data for projected volume kube-api-access-wvmqz for pod openshift-network-diagnostics/network-check-target-t4vsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:24.507551 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:24.507526    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz podName:9ceed3fb-b91c-43c2-8813-7991a3c27659 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:28.507511202 +0000 UTC m=+10.074796179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvmqz" (UniqueName: "kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz") pod "network-check-target-t4vsn" (UID: "9ceed3fb-b91c-43c2-8813-7991a3c27659") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:24.507551 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:24.507369    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:24.507868 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:24.507565    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret podName:c5f3eafe-b7a2-47e9-954a-028746276935 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:28.507554761 +0000 UTC m=+10.074839725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret") pod "global-pull-secret-syncer-q89rm" (UID: "c5f3eafe-b7a2-47e9-954a-028746276935") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:24.816269 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:24.816187    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:24.816443 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:24.816187    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:24.816443 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:24.816319    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:24.816443 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:24.816416    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:24.816558 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:24.816480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:24.816597 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:24.816558    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:24.864045 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:24.863957    2578 generic.go:356] "Generic (PLEG): container finished" podID="080a0830f4e17cb8b7f0213f1a606519" containerID="7c07945c9f4ca08f08b7313170bc9d91b56876f47ada82ac09ddb7100369349a" exitCode=0
Jul 01 18:06:24.864045 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:24.864000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal" event={"ID":"080a0830f4e17cb8b7f0213f1a606519","Type":"ContainerDied","Data":"7c07945c9f4ca08f08b7313170bc9d91b56876f47ada82ac09ddb7100369349a"}
Jul 01 18:06:26.816010 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:26.815978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:26.816487 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:26.816151    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:26.816487 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:26.816158    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:26.816487 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:26.816238    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:26.816487 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:26.816275    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:26.816487 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:26.816325    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:28.540771 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:28.540730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:28.541251 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:28.540796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqz\" (UniqueName: \"kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz\") pod \"network-check-target-t4vsn\" (UID: \"9ceed3fb-b91c-43c2-8813-7991a3c27659\") " pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:28.541251 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:28.540835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:28.541251 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:28.540990    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:28.541251 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:28.541076    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret podName:c5f3eafe-b7a2-47e9-954a-028746276935 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:36.541053048 +0000 UTC m=+18.108338023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret") pod "global-pull-secret-syncer-q89rm" (UID: "c5f3eafe-b7a2-47e9-954a-028746276935") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:28.541548 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:28.541525    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:28.541605 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:28.541557    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:06:28.541605 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:28.541575    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:06:28.541605 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:28.541589    2578 projected.go:196] Error preparing data for projected volume kube-api-access-wvmqz for pod openshift-network-diagnostics/network-check-target-t4vsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:28.541605 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:28.541595    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs podName:8936baa9-7b61-4534-b564-a7bb12bf3535 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:36.541577875 +0000 UTC m=+18.108862848 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs") pod "network-metrics-daemon-2r8hd" (UID: "8936baa9-7b61-4534-b564-a7bb12bf3535") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:28.541822 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:28.541628    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz podName:9ceed3fb-b91c-43c2-8813-7991a3c27659 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:36.541617928 +0000 UTC m=+18.108902900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvmqz" (UniqueName: "kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz") pod "network-check-target-t4vsn" (UID: "9ceed3fb-b91c-43c2-8813-7991a3c27659") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:28.815206 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:28.815125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:28.815382 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:28.815249    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:28.817610 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:28.817575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:28.817880 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:28.817853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:28.818781 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:28.818694    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:28.818898 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:28.818778    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:30.815923 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:30.815890    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:30.815923 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:30.815923    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:30.816435 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:30.816088    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:30.816435 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:30.816156    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:30.816435 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:30.816171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:30.816435 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:30.816242    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:32.818180 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:32.818149    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:32.818611 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:32.818256    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:32.818611 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:32.818269    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:32.818611 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:32.818278    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:32.818611 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:32.818399    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:32.818611 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:32.818481    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:34.818630 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:34.818604    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:34.819096 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:34.818599    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:34.819096 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:34.818708    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:34.819096 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:34.818818    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:34.819096 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:34.818608    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:34.819096 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:34.818920    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:36.605522 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:36.605492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqz\" (UniqueName: \"kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz\") pod \"network-check-target-t4vsn\" (UID: \"9ceed3fb-b91c-43c2-8813-7991a3c27659\") " pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:36.606098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:36.605540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:36.606098 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:36.605598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:36.606098 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:36.605690    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:06:36.606098 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:36.605701    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:36.606098 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:36.605707    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:36.606098 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:36.605710    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:06:36.606098 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:36.605759    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs podName:8936baa9-7b61-4534-b564-a7bb12bf3535 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:52.605739236 +0000 UTC m=+34.173024198 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs") pod "network-metrics-daemon-2r8hd" (UID: "8936baa9-7b61-4534-b564-a7bb12bf3535") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:36.606098 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:36.605764    2578 projected.go:196] Error preparing data for projected volume kube-api-access-wvmqz for pod openshift-network-diagnostics/network-check-target-t4vsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:36.606098 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:36.605777    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret podName:c5f3eafe-b7a2-47e9-954a-028746276935 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:52.605767693 +0000 UTC m=+34.173052653 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret") pod "global-pull-secret-syncer-q89rm" (UID: "c5f3eafe-b7a2-47e9-954a-028746276935") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:36.606098 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:36.605805    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz podName:9ceed3fb-b91c-43c2-8813-7991a3c27659 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:52.605788966 +0000 UTC m=+34.173073925 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvmqz" (UniqueName: "kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz") pod "network-check-target-t4vsn" (UID: "9ceed3fb-b91c-43c2-8813-7991a3c27659") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:36.818256 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:36.818227    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:36.818256 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:36.818253    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:36.818482 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:36.818339    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:36.818482 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:36.818387    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:36.818482 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:36.818467    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:36.818680 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:36.818541    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:38.815822 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:38.815790    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:38.816240 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:38.815889    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:38.816240 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:38.815902    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:38.816240 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:38.815980    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:38.816240 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:38.816103    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:38.816240 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:38.816196    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:40.815905 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.815523    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:40.816647 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.815528    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:40.816647 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.815524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:40.816647 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:40.816015    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:40.816647 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:40.816070    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:40.816647 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:40.816152    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:40.894743 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.894707    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bbppr" event={"ID":"9d1e4c3c-8594-4a71-86fe-9e77d4f4153f","Type":"ContainerStarted","Data":"ca077d0a67e415feba7c328f6d70e0d8f4b3fb153abefcea825f749513cc557c"}
Jul 01 18:06:40.896773 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.896745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rc8bn" event={"ID":"65be3311-bd63-439e-9980-ff65266cef62","Type":"ContainerStarted","Data":"5e9d5f85013d6f7663d790e3d8795f6f00d864e96b5999093a45fc304b038d0c"}
Jul 01 18:06:40.899495 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.899394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hmcsv" event={"ID":"4ce225ff-82c4-49f3-9dd0-b8df3d1b196d","Type":"ContainerStarted","Data":"00d1993b8ee68ed1a841edfbb86d65480cb0a9c235875d84b29a64c83da4405d"}
Jul 01 18:06:40.902490 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.902437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal" event={"ID":"080a0830f4e17cb8b7f0213f1a606519","Type":"ContainerStarted","Data":"22e56ce2cd3dc08c14a48d813621c8220075b2b8bb884e2949adc2d72ee9e0de"}
Jul 01 18:06:40.904270 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.904217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679" event={"ID":"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce","Type":"ContainerStarted","Data":"6e677cf3523bb22298295893c68236ebf70570a07fd57149f1fa5ecaa54bda07"}
Jul 01 18:06:40.905937 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.905839    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-67lbw" event={"ID":"a7450108-384d-45a2-b88d-ac61a546bab4","Type":"ContainerStarted","Data":"ec6f7f2a7726723198363585fc0359d84e424c5fa35f07fb7fbf606d8c0e7bef"}
Jul 01 18:06:40.907479 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.907454    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x6rq6" event={"ID":"f7223ef7-a5f6-4690-be25-9bed08246516","Type":"ContainerStarted","Data":"25500fdb11d148f649bda22378ff5053363f6e650ccbdd676780187c5d4282dd"}
Jul 01 18:06:40.910815 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.910762    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bbppr" podStartSLOduration=2.830735954 podStartE2EDuration="21.910746456s" totalImagesPullingTime="19.080010502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:40.909501654 +0000 UTC m=+22.476786658" watchObservedRunningTime="2026-07-01 18:06:40.910746456 +0000 UTC m=+22.478031436"
Jul 01 18:06:40.912036 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.912015    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:06:40.912402 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.912377    2578 generic.go:356] "Generic (PLEG): container finished" podID="11a3da3b-b7dc-4aed-b261-597c2fc14fdb" containerID="28df6b8119840e52c794e95308ffd410682554605b3c04c6879c7178f0314390" exitCode=1
Jul 01 18:06:40.912493 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.912478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn" event={"ID":"11a3da3b-b7dc-4aed-b261-597c2fc14fdb","Type":"ContainerStarted","Data":"c7a16bf5bbde82aa769ba940e68b1d811b9f91dcc074a0d4a5cce0e701c9fd10"}
Jul 01 18:06:40.912612 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.912597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn" event={"ID":"11a3da3b-b7dc-4aed-b261-597c2fc14fdb","Type":"ContainerStarted","Data":"e3b00169aa14a37f04f385730def7817152b60df58b4e1eb4b1ab7ab5cab95e9"}
Jul 01 18:06:40.912676 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.912660    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn" event={"ID":"11a3da3b-b7dc-4aed-b261-597c2fc14fdb","Type":"ContainerStarted","Data":"16b2c94b7f49556d89906743843b99864f8c9771e315d328cf604a4419cc2cfa"}
Jul 01 18:06:40.912764 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.912740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn" event={"ID":"11a3da3b-b7dc-4aed-b261-597c2fc14fdb","Type":"ContainerStarted","Data":"cc8fc546261024c0a72d4f2dcad970fe933b0fb8b8d2d9741c15951cc943aba0"}
Jul 01 18:06:40.912958 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.912921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn" event={"ID":"11a3da3b-b7dc-4aed-b261-597c2fc14fdb","Type":"ContainerDied","Data":"28df6b8119840e52c794e95308ffd410682554605b3c04c6879c7178f0314390"}
Jul 01 18:06:40.913360 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.913336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn" event={"ID":"11a3da3b-b7dc-4aed-b261-597c2fc14fdb","Type":"ContainerStarted","Data":"3dbfd6839be329400e79734646f8f7030873d97bd3b5bacc73c99e1a56f35c64"}
Jul 01 18:06:40.914653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.914632    2578 generic.go:356] "Generic (PLEG): container finished" podID="298d79e6-cc04-47e5-98a5-36e2284c5174" containerID="c6b5c072fa07dac4bf9a3056277ae9e02f1b26efa631446dfc14fed40c8e0504" exitCode=0
Jul 01 18:06:40.914760 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.914663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg6np" event={"ID":"298d79e6-cc04-47e5-98a5-36e2284c5174","Type":"ContainerDied","Data":"c6b5c072fa07dac4bf9a3056277ae9e02f1b26efa631446dfc14fed40c8e0504"}
Jul 01 18:06:40.926957 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.926815    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rc8bn" podStartSLOduration=2.782150498 podStartE2EDuration="21.926796231s" totalImagesPullingTime="19.144645733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:40.926784415 +0000 UTC m=+22.494069395" watchObservedRunningTime="2026-07-01 18:06:40.926796231 +0000 UTC m=+22.494081211"
Jul 01 18:06:40.941578 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.941534    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x6rq6" podStartSLOduration=2.8907563830000003 podStartE2EDuration="21.941519122s" totalImagesPullingTime="19.050762739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:40.941417422 +0000 UTC m=+22.508702402" watchObservedRunningTime="2026-07-01 18:06:40.941519122 +0000 UTC m=+22.508804100"
Jul 01 18:06:40.959302 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.959255    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-67lbw" podStartSLOduration=2.915363901 podStartE2EDuration="21.959236827s" totalImagesPullingTime="19.043872926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:40.958604893 +0000 UTC m=+22.525889873" watchObservedRunningTime="2026-07-01 18:06:40.959236827 +0000 UTC m=+22.526521832"
Jul 01 18:06:40.972841 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.972798    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hmcsv" podStartSLOduration=4.262522708 podStartE2EDuration="21.972783766s" totalImagesPullingTime="17.710261058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:40.972680725 +0000 UTC m=+22.539965705" watchObservedRunningTime="2026-07-01 18:06:40.972783766 +0000 UTC m=+22.540068745"
Jul 01 18:06:40.996939 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:40.996838    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-161.ec2.internal" podStartSLOduration=20.996817785 podStartE2EDuration="20.996817785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:40.996061153 +0000 UTC m=+22.563346131" watchObservedRunningTime="2026-07-01 18:06:40.996817785 +0000 UTC m=+22.564102764"
Jul 01 18:06:41.748477 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:41.748449    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 18:06:41.825751 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:41.825600    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T18:06:41.748470578Z","UUID":"e9458ff3-8d66-4786-963c-cdbc80252928","Handler":null,"Name":"","Endpoint":""}
Jul 01 18:06:41.827233 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:41.827209    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 18:06:41.827340 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:41.827245    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 18:06:41.918383 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:41.918352    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679" event={"ID":"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce","Type":"ContainerStarted","Data":"2bb64484a832cdeeaa09faa9b3dd00b27f0262225dd19ee73ea69b7853f1a609"}
Jul 01 18:06:41.919862 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:41.919835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zgwh2" event={"ID":"583c6fb3-3031-43f8-bce9-aa90d151a40b","Type":"ContainerStarted","Data":"dc241a07a7961d59f897931b1ab74741a061649be8cf8984b8909d721ce90c71"}
Jul 01 18:06:41.935477 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:41.935432    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zgwh2" podStartSLOduration=12.910326057 podStartE2EDuration="22.935419012s" totalImagesPullingTime="10.025092955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:41.935383736 +0000 UTC m=+23.502668715" watchObservedRunningTime="2026-07-01 18:06:41.935419012 +0000 UTC m=+23.502703990"
Jul 01 18:06:42.818843 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:42.818816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:42.818966 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:42.818923    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:42.818966 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:42.818938    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:42.819127 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:42.819039    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:42.819127 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:42.819090    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:42.819232 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:42.819170    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:42.924564 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:42.924470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679" event={"ID":"dcc9a1bf-40a4-4775-9eef-fcc8e52257ce","Type":"ContainerStarted","Data":"7fdf501a20e9d9d3bba5d13bd158abf0cd777bf1cb8c70002c63f9321ab97cb8"}
Jul 01 18:06:42.942088 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:42.941858    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br679" podStartSLOduration=10.924693312 podStartE2EDuration="23.941840618s" totalImagesPullingTime="13.017147306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:42.941396355 +0000 UTC m=+24.508681336" watchObservedRunningTime="2026-07-01 18:06:42.941840618 +0000 UTC m=+24.509125598"
Jul 01 18:06:43.930378 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:43.930353    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:06:43.930825 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:43.930780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn" event={"ID":"11a3da3b-b7dc-4aed-b261-597c2fc14fdb","Type":"ContainerStarted","Data":"fb2d42393ef48745c14e5541870834a8ce057a2826559e261deffeaef121f8ce"}
Jul 01 18:06:44.815441 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:44.815410    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:44.815614 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:44.815410    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:44.815614 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:44.815418    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:44.815614 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:44.815544    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:44.815772 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:44.815618    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:44.815772 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:44.815742    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:45.277618 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:45.277589    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x6rq6"
Jul 01 18:06:45.278493 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:45.278474    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x6rq6"
Jul 01 18:06:45.939028 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:45.938808    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:06:45.939487 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:45.939459    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn" event={"ID":"11a3da3b-b7dc-4aed-b261-597c2fc14fdb","Type":"ContainerStarted","Data":"46b8cfea81a35e131cdd0d10df04ceadaed289776cbf1420998a6f1a146138e7"}
Jul 01 18:06:45.939884 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:45.939748    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:45.939884 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:45.939833    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:45.940070 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:45.940015    2578 scope.go:117] "RemoveContainer" containerID="28df6b8119840e52c794e95308ffd410682554605b3c04c6879c7178f0314390"
Jul 01 18:06:45.941697 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:45.941671    2578 generic.go:356] "Generic (PLEG): container finished" podID="298d79e6-cc04-47e5-98a5-36e2284c5174" containerID="64f3e66daa90133c209ab57209a9864ebfdedca731d232bc9afd20329e6bdaba" exitCode=0
Jul 01 18:06:45.941808 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:45.941746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg6np" event={"ID":"298d79e6-cc04-47e5-98a5-36e2284c5174","Type":"ContainerDied","Data":"64f3e66daa90133c209ab57209a9864ebfdedca731d232bc9afd20329e6bdaba"}
Jul 01 18:06:45.942048 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:45.942018    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x6rq6"
Jul 01 18:06:45.942754 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:45.942647    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x6rq6"
Jul 01 18:06:45.956812 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:45.956792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:46.815621 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:46.815591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:46.816122 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:46.815591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:46.816122 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:46.815701    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:46.816122 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:46.815715    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:46.816122 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:46.815819    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:46.816122 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:46.815899    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:46.948222 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:46.948198    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:06:46.948604 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:46.948579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn" event={"ID":"11a3da3b-b7dc-4aed-b261-597c2fc14fdb","Type":"ContainerStarted","Data":"b16f3bc42b18bd368142d8fd018881c1e1ac1b48880261acd2e6de8824164699"}
Jul 01 18:06:46.948904 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:46.948867    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:46.963220 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:46.963202    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:06:46.976655 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:46.976618    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn" podStartSLOduration=8.853248991 podStartE2EDuration="27.976606185s" totalImagesPullingTime="19.123357194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:46.975094502 +0000 UTC m=+28.542379482" watchObservedRunningTime="2026-07-01 18:06:46.976606185 +0000 UTC m=+28.543891173"
Jul 01 18:06:47.322491 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:47.322462    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t4vsn"]
Jul 01 18:06:47.322600 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:47.322552    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:47.322639 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:47.322625    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:47.325391 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:47.325362    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q89rm"]
Jul 01 18:06:47.325499 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:47.325442    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:47.325561 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:47.325519    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:47.326082 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:47.326060    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2r8hd"]
Jul 01 18:06:47.326168 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:47.326154    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:47.326264 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:47.326243    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:47.952307 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:47.952272    2578 generic.go:356] "Generic (PLEG): container finished" podID="298d79e6-cc04-47e5-98a5-36e2284c5174" containerID="7d88fba26543d2da3a50254866fc45351125103a6692154ce240e08dc549ad88" exitCode=0
Jul 01 18:06:47.952665 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:47.952344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg6np" event={"ID":"298d79e6-cc04-47e5-98a5-36e2284c5174","Type":"ContainerDied","Data":"7d88fba26543d2da3a50254866fc45351125103a6692154ce240e08dc549ad88"}
Jul 01 18:06:48.816049 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:48.816019    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:48.816233 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:48.816063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:48.816233 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:48.816126    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:48.816233 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:48.816190    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:48.816477 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:48.816189    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:48.816477 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:48.816283    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:49.958390 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:49.958358    2578 generic.go:356] "Generic (PLEG): container finished" podID="298d79e6-cc04-47e5-98a5-36e2284c5174" containerID="b1751cd14b8c1ad88ef10c184cff87c4ca0a821af3303974506719b189617378" exitCode=0
Jul 01 18:06:49.959112 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:49.958408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg6np" event={"ID":"298d79e6-cc04-47e5-98a5-36e2284c5174","Type":"ContainerDied","Data":"b1751cd14b8c1ad88ef10c184cff87c4ca0a821af3303974506719b189617378"}
Jul 01 18:06:50.815591 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:50.815556    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:50.815768 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:50.815564    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:50.815768 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:50.815682    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:50.815768 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:50.815692    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:50.815932 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:50.815762    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:50.815932 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:50.815854    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:52.640097 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:52.640059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:52.640667 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:52.640116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqz\" (UniqueName: \"kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz\") pod \"network-check-target-t4vsn\" (UID: \"9ceed3fb-b91c-43c2-8813-7991a3c27659\") " pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:52.640667 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:52.640153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:52.640667 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:52.640224    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:52.640667 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:52.640292    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs podName:8936baa9-7b61-4534-b564-a7bb12bf3535 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:24.640271468 +0000 UTC m=+66.207556426 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs") pod "network-metrics-daemon-2r8hd" (UID: "8936baa9-7b61-4534-b564-a7bb12bf3535") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:52.640667 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:52.640313    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:06:52.640667 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:52.640335    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:06:52.640667 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:52.640337    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:52.640667 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:52.640347    2578 projected.go:196] Error preparing data for projected volume kube-api-access-wvmqz for pod openshift-network-diagnostics/network-check-target-t4vsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:52.640667 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:52.640388    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret podName:c5f3eafe-b7a2-47e9-954a-028746276935 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:24.640376087 +0000 UTC m=+66.207661043 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret") pod "global-pull-secret-syncer-q89rm" (UID: "c5f3eafe-b7a2-47e9-954a-028746276935") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:52.640667 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:52.640401    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz podName:9ceed3fb-b91c-43c2-8813-7991a3c27659 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:24.640395428 +0000 UTC m=+66.207680386 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvmqz" (UniqueName: "kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz") pod "network-check-target-t4vsn" (UID: "9ceed3fb-b91c-43c2-8813-7991a3c27659") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:52.815862 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:52.815830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:52.815862 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:52.815854    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:52.816099 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:52.815832    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:52.816099 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:52.815943    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q89rm" podUID="c5f3eafe-b7a2-47e9-954a-028746276935"
Jul 01 18:06:52.816099 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:52.816042    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4vsn" podUID="9ceed3fb-b91c-43c2-8813-7991a3c27659"
Jul 01 18:06:52.816237 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:52.816124    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:06:53.282287 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.282259    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-161.ec2.internal" event="NodeReady"
Jul 01 18:06:53.282441 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.282394    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 18:06:53.323483 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.323456    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55bdccbb57-zl6tl"]
Jul 01 18:06:53.328266 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.328237    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"]
Jul 01 18:06:53.328772 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.328473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.330908 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.330888    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-86d4l\""
Jul 01 18:06:53.331259 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.330987    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 01 18:06:53.331537 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.331510    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 01 18:06:53.331616 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.331548    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 01 18:06:53.332324 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.332305    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk"]
Jul 01 18:06:53.332469 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.332446    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.335872 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.335713    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 01 18:06:53.335957 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.335929    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 01 18:06:53.336086 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.335929    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 01 18:06:53.336144 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.336090    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 01 18:06:53.336201 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.336159    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 01 18:06:53.336360 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.336336    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"]
Jul 01 18:06:53.336459 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.336428    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 01 18:06:53.336540 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.336521    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 01 18:06:53.336660 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.336542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk"
Jul 01 18:06:53.338287 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.338270    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 01 18:06:53.339818 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.339791    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"]
Jul 01 18:06:53.339940 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.339921    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 01 18:06:53.340151 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.340133    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7tkpn\""
Jul 01 18:06:53.340252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.340234    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:06:53.343292 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.343271    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fnvvx\""
Jul 01 18:06:53.343292 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.343287    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 01 18:06:53.343816 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.343798    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"]
Jul 01 18:06:53.343906 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.343830    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dhrff"]
Jul 01 18:06:53.343967 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.343940    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 01 18:06:53.344227 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.344193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:06:53.344837 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.344818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:06:53.344956 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.344851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc0e33ec-c57e-43f2-97cc-28b7a97db026-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.344956 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.344896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:06:53.344956 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.344946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-image-registry-private-configuration\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.345173 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.344969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-certificates\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.345173 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.345011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fc0e33ec-c57e-43f2-97cc-28b7a97db026-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.345173 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.345043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-bound-sa-token\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.345173 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.345071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-trusted-ca\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.345173 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.345099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-ca-trust-extracted\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.345173 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.345122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-installation-pull-secrets\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.345173 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.345148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdfwm\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-kube-api-access-jdfwm\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.345173 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.345173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fc0e33ec-c57e-43f2-97cc-28b7a97db026-hub\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.346241 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.345196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5khr7\" (UniqueName: \"kubernetes.io/projected/fc0e33ec-c57e-43f2-97cc-28b7a97db026-kube-api-access-5khr7\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.346241 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.345221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fc0e33ec-c57e-43f2-97cc-28b7a97db026-ca\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.346241 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.345243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0e33ec-c57e-43f2-97cc-28b7a97db026-additional-service-ca\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.346241 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.345274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc4e980d-eaf3-4a0b-86b8-c94656918bbd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-768c67f45d-2mbtk\" (UID: \"dc4e980d-eaf3-4a0b-86b8-c94656918bbd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk"
Jul 01 18:06:53.346241 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.345320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwvcw\" (UniqueName: \"kubernetes.io/projected/dc4e980d-eaf3-4a0b-86b8-c94656918bbd-kube-api-access-nwvcw\") pod \"managed-serviceaccount-addon-agent-768c67f45d-2mbtk\" (UID: \"dc4e980d-eaf3-4a0b-86b8-c94656918bbd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk"
Jul 01 18:06:53.346241 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.345345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.348101 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.348081    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 01 18:06:53.350226 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.350206    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk"]
Jul 01 18:06:53.350317 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.350234    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55bdccbb57-zl6tl"]
Jul 01 18:06:53.350317 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.350255    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b7ls7"]
Jul 01 18:06:53.350424 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.350315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dhrff"
Jul 01 18:06:53.352589 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.352570    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 18:06:53.352959 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.352939    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 18:06:53.353345 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.353028    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cl7bm\""
Jul 01 18:06:53.353345 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.353141    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 18:06:53.354136 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.353649    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:53.354818 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.354759    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"]
Jul 01 18:06:53.355661 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.355637    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dhrff"]
Jul 01 18:06:53.356022 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.355946    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 18:06:53.356127 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.356110    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 18:06:53.356215 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.356191    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vsbfc\""
Jul 01 18:06:53.357388 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.357371    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"]
Jul 01 18:06:53.361047 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.361032    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b7ls7"]
Jul 01 18:06:53.446370 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-ca-trust-extracted\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.446475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61840d9c-093c-4b02-8abe-e1db22856cbf-config-volume\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:53.446475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5khr7\" (UniqueName: \"kubernetes.io/projected/fc0e33ec-c57e-43f2-97cc-28b7a97db026-kube-api-access-5khr7\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.446475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fc0e33ec-c57e-43f2-97cc-28b7a97db026-ca\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.446475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0e33ec-c57e-43f2-97cc-28b7a97db026-additional-service-ca\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.446475 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwvcw\" (UniqueName: \"kubernetes.io/projected/dc4e980d-eaf3-4a0b-86b8-c94656918bbd-kube-api-access-nwvcw\") pod \"managed-serviceaccount-addon-agent-768c67f45d-2mbtk\" (UID: \"dc4e980d-eaf3-4a0b-86b8-c94656918bbd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk"
Jul 01 18:06:53.446728 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4mgg\" (UniqueName: \"kubernetes.io/projected/61840d9c-093c-4b02-8abe-e1db22856cbf-kube-api-access-z4mgg\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:53.446728 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc0e33ec-c57e-43f2-97cc-28b7a97db026-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.446728 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:06:53.446728 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-image-registry-private-configuration\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.446728 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-installation-pull-secrets\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.446728 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61840d9c-093c-4b02-8abe-e1db22856cbf-tmp-dir\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:53.446728 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fc0e33ec-c57e-43f2-97cc-28b7a97db026-hub\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.446728 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc4e980d-eaf3-4a0b-86b8-c94656918bbd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-768c67f45d-2mbtk\" (UID: \"dc4e980d-eaf3-4a0b-86b8-c94656918bbd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk"
Jul 01 18:06:53.446728 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/15516c4a-48cf-4121-9e72-6303618ee567-klusterlet-config\") pod \"klusterlet-addon-workmgr-594b6c4d4d-fmr7p\" (UID: \"15516c4a-48cf-4121-9e72-6303618ee567\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:06:53.447168 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15516c4a-48cf-4121-9e72-6303618ee567-tmp\") pod \"klusterlet-addon-workmgr-594b6c4d4d-fmr7p\" (UID: \"15516c4a-48cf-4121-9e72-6303618ee567\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:06:53.447168 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446764    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-ca-trust-extracted\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.447168 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.447168 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcz5l\" (UniqueName: \"kubernetes.io/projected/15516c4a-48cf-4121-9e72-6303618ee567-kube-api-access-gcz5l\") pod \"klusterlet-addon-workmgr-594b6c4d4d-fmr7p\" (UID: \"15516c4a-48cf-4121-9e72-6303618ee567\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:06:53.447168 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65kwp\" (UniqueName: \"kubernetes.io/projected/fcdd9ed9-9c31-4499-8ace-63a0ec626420-kube-api-access-65kwp\") pod \"ingress-canary-dhrff\" (UID: \"fcdd9ed9-9c31-4499-8ace-63a0ec626420\") " pod="openshift-ingress-canary/ingress-canary-dhrff"
Jul 01 18:06:53.447168 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:53.447168 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:06:53.447168 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-bound-sa-token\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.447168 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcdd9ed9-9c31-4499-8ace-63a0ec626420-cert\") pod \"ingress-canary-dhrff\" (UID: \"fcdd9ed9-9c31-4499-8ace-63a0ec626420\") " pod="openshift-ingress-canary/ingress-canary-dhrff"
Jul 01 18:06:53.447168 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.446970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-certificates\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.447168 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.447010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fc0e33ec-c57e-43f2-97cc-28b7a97db026-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.447168 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.447039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdfwm\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-kube-api-access-jdfwm\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.447168 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:53.447108    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:06:53.447168 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:53.447126    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bdccbb57-zl6tl: secret "image-registry-tls" not found
Jul 01 18:06:53.448216 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:53.447192    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls podName:58c018bb-8a6f-4e8f-8a94-51f9bf03a063 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:53.947172583 +0000 UTC m=+35.514457553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls") pod "image-registry-55bdccbb57-zl6tl" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063") : secret "image-registry-tls" not found
Jul 01 18:06:53.448216 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:53.447483    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 18:06:53.448216 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:53.447560    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert podName:4c69cbc1-40fc-4abc-909b-bd59ced4b2a2 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:53.947541873 +0000 UTC m=+35.514826844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-jm2m4" (UID: "4c69cbc1-40fc-4abc-909b-bd59ced4b2a2") : secret "networking-console-plugin-cert" not found
Jul 01 18:06:53.448216 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.447903    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0e33ec-c57e-43f2-97cc-28b7a97db026-additional-service-ca\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.448916 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.448846    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-certificates\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.449022 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.448957    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:06:53.449091 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.449037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-trusted-ca\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.449950 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.449922    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-trusted-ca\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.451935 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.451833    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fc0e33ec-c57e-43f2-97cc-28b7a97db026-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.451935 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.451837    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fc0e33ec-c57e-43f2-97cc-28b7a97db026-hub\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.451935 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.451874    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc0e33ec-c57e-43f2-97cc-28b7a97db026-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.452264 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.452220    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fc0e33ec-c57e-43f2-97cc-28b7a97db026-ca\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.452264 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.452248    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc4e980d-eaf3-4a0b-86b8-c94656918bbd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-768c67f45d-2mbtk\" (UID: \"dc4e980d-eaf3-4a0b-86b8-c94656918bbd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk"
Jul 01 18:06:53.453289 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.453268    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-image-registry-private-configuration\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.454258 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.453880    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-installation-pull-secrets\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.456119 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.456097    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwvcw\" (UniqueName: \"kubernetes.io/projected/dc4e980d-eaf3-4a0b-86b8-c94656918bbd-kube-api-access-nwvcw\") pod \"managed-serviceaccount-addon-agent-768c67f45d-2mbtk\" (UID: \"dc4e980d-eaf3-4a0b-86b8-c94656918bbd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk"
Jul 01 18:06:53.456484 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.456461    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5khr7\" (UniqueName: \"kubernetes.io/projected/fc0e33ec-c57e-43f2-97cc-28b7a97db026-kube-api-access-5khr7\") pod \"cluster-proxy-proxy-agent-774584c764-kkjr6\" (UID: \"fc0e33ec-c57e-43f2-97cc-28b7a97db026\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.456584 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.456494    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-bound-sa-token\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.456644 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.456613    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdfwm\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-kube-api-access-jdfwm\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.549548 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.549477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61840d9c-093c-4b02-8abe-e1db22856cbf-tmp-dir\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:53.549548 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.549526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/15516c4a-48cf-4121-9e72-6303618ee567-klusterlet-config\") pod \"klusterlet-addon-workmgr-594b6c4d4d-fmr7p\" (UID: \"15516c4a-48cf-4121-9e72-6303618ee567\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:06:53.549751 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.549663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15516c4a-48cf-4121-9e72-6303618ee567-tmp\") pod \"klusterlet-addon-workmgr-594b6c4d4d-fmr7p\" (UID: \"15516c4a-48cf-4121-9e72-6303618ee567\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:06:53.549751 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.549716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcz5l\" (UniqueName: \"kubernetes.io/projected/15516c4a-48cf-4121-9e72-6303618ee567-kube-api-access-gcz5l\") pod \"klusterlet-addon-workmgr-594b6c4d4d-fmr7p\" (UID: \"15516c4a-48cf-4121-9e72-6303618ee567\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:06:53.549834 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.549758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65kwp\" (UniqueName: \"kubernetes.io/projected/fcdd9ed9-9c31-4499-8ace-63a0ec626420-kube-api-access-65kwp\") pod \"ingress-canary-dhrff\" (UID: \"fcdd9ed9-9c31-4499-8ace-63a0ec626420\") " pod="openshift-ingress-canary/ingress-canary-dhrff"
Jul 01 18:06:53.549834 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.549782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:53.549932 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.549834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcdd9ed9-9c31-4499-8ace-63a0ec626420-cert\") pod \"ingress-canary-dhrff\" (UID: \"fcdd9ed9-9c31-4499-8ace-63a0ec626420\") " pod="openshift-ingress-canary/ingress-canary-dhrff"
Jul 01 18:06:53.549932 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.549876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61840d9c-093c-4b02-8abe-e1db22856cbf-config-volume\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:53.549932 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.549889    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61840d9c-093c-4b02-8abe-e1db22856cbf-tmp-dir\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:53.549932 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.549915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4mgg\" (UniqueName: \"kubernetes.io/projected/61840d9c-093c-4b02-8abe-e1db22856cbf-kube-api-access-z4mgg\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:53.550133 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:53.549944    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:06:53.550133 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:53.550038    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls podName:61840d9c-093c-4b02-8abe-e1db22856cbf nodeName:}" failed. No retries permitted until 2026-07-01 18:06:54.050018127 +0000 UTC m=+35.617303106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls") pod "dns-default-b7ls7" (UID: "61840d9c-093c-4b02-8abe-e1db22856cbf") : secret "dns-default-metrics-tls" not found
Jul 01 18:06:53.550133 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.550062    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15516c4a-48cf-4121-9e72-6303618ee567-tmp\") pod \"klusterlet-addon-workmgr-594b6c4d4d-fmr7p\" (UID: \"15516c4a-48cf-4121-9e72-6303618ee567\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:06:53.550568 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.550541    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61840d9c-093c-4b02-8abe-e1db22856cbf-config-volume\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:53.552404 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.552383    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/15516c4a-48cf-4121-9e72-6303618ee567-klusterlet-config\") pod \"klusterlet-addon-workmgr-594b6c4d4d-fmr7p\" (UID: \"15516c4a-48cf-4121-9e72-6303618ee567\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:06:53.552557 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.552540    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcdd9ed9-9c31-4499-8ace-63a0ec626420-cert\") pod \"ingress-canary-dhrff\" (UID: \"fcdd9ed9-9c31-4499-8ace-63a0ec626420\") " pod="openshift-ingress-canary/ingress-canary-dhrff"
Jul 01 18:06:53.560891 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.560868    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcz5l\" (UniqueName: \"kubernetes.io/projected/15516c4a-48cf-4121-9e72-6303618ee567-kube-api-access-gcz5l\") pod \"klusterlet-addon-workmgr-594b6c4d4d-fmr7p\" (UID: \"15516c4a-48cf-4121-9e72-6303618ee567\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:06:53.561891 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.561869    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4mgg\" (UniqueName: \"kubernetes.io/projected/61840d9c-093c-4b02-8abe-e1db22856cbf-kube-api-access-z4mgg\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:53.562388 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.562370    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65kwp\" (UniqueName: \"kubernetes.io/projected/fcdd9ed9-9c31-4499-8ace-63a0ec626420-kube-api-access-65kwp\") pod \"ingress-canary-dhrff\" (UID: \"fcdd9ed9-9c31-4499-8ace-63a0ec626420\") " pod="openshift-ingress-canary/ingress-canary-dhrff"
Jul 01 18:06:53.658770 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.658746    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:06:53.667310 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.667156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk"
Jul 01 18:06:53.688516 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.688487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:06:53.694533 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.694507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dhrff"
Jul 01 18:06:53.953582 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.953550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:53.953720 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:53.953604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:06:53.953800 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:53.953747    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 18:06:53.953842 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:53.953819    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert podName:4c69cbc1-40fc-4abc-909b-bd59ced4b2a2 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:54.953805418 +0000 UTC m=+36.521090376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-jm2m4" (UID: "4c69cbc1-40fc-4abc-909b-bd59ced4b2a2") : secret "networking-console-plugin-cert" not found
Jul 01 18:06:53.954063 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:53.954023    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:06:53.954063 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:53.954055    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bdccbb57-zl6tl: secret "image-registry-tls" not found
Jul 01 18:06:53.954259 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:53.954124    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls podName:58c018bb-8a6f-4e8f-8a94-51f9bf03a063 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:54.954107369 +0000 UTC m=+36.521392343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls") pod "image-registry-55bdccbb57-zl6tl" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063") : secret "image-registry-tls" not found
Jul 01 18:06:54.054738 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.054654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:54.054909 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:54.054850    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:06:54.054968 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:54.054912    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls podName:61840d9c-093c-4b02-8abe-e1db22856cbf nodeName:}" failed. No retries permitted until 2026-07-01 18:06:55.054893783 +0000 UTC m=+36.622178756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls") pod "dns-default-b7ls7" (UID: "61840d9c-093c-4b02-8abe-e1db22856cbf") : secret "dns-default-metrics-tls" not found
Jul 01 18:06:54.074061 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.074039    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dhrff"]
Jul 01 18:06:54.077854 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:54.077829    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdd9ed9_9c31_4499_8ace_63a0ec626420.slice/crio-ad8868d850b458955591d9bbc71503cfc7397aaf84c39dc86546de5c0d18b62b WatchSource:0}: Error finding container ad8868d850b458955591d9bbc71503cfc7397aaf84c39dc86546de5c0d18b62b: Status 404 returned error can't find the container with id ad8868d850b458955591d9bbc71503cfc7397aaf84c39dc86546de5c0d18b62b
Jul 01 18:06:54.081249 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.081216    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"]
Jul 01 18:06:54.081974 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.081955    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"]
Jul 01 18:06:54.088050 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:54.088020    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15516c4a_48cf_4121_9e72_6303618ee567.slice/crio-8a1fb5bc86a0c5ccceb965601928d5c086a1df0e5cfff60074c93d9a9c2ae6c8 WatchSource:0}: Error finding container 8a1fb5bc86a0c5ccceb965601928d5c086a1df0e5cfff60074c93d9a9c2ae6c8: Status 404 returned error can't find the container with id 8a1fb5bc86a0c5ccceb965601928d5c086a1df0e5cfff60074c93d9a9c2ae6c8
Jul 01 18:06:54.088279 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:54.088261    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc0e33ec_c57e_43f2_97cc_28b7a97db026.slice/crio-382f742f82b23c1aa161425e5e27f61c7a67ac6140affec3ef746d5f2c4a9796 WatchSource:0}: Error finding container 382f742f82b23c1aa161425e5e27f61c7a67ac6140affec3ef746d5f2c4a9796: Status 404 returned error can't find the container with id 382f742f82b23c1aa161425e5e27f61c7a67ac6140affec3ef746d5f2c4a9796
Jul 01 18:06:54.090697 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.090677    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk"]
Jul 01 18:06:54.093678 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:06:54.093661    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc4e980d_eaf3_4a0b_86b8_c94656918bbd.slice/crio-4e5f0f028df878dbd4c50764560a9f34b2d7b8c1c546ce5a29163fdad6258645 WatchSource:0}: Error finding container 4e5f0f028df878dbd4c50764560a9f34b2d7b8c1c546ce5a29163fdad6258645: Status 404 returned error can't find the container with id 4e5f0f028df878dbd4c50764560a9f34b2d7b8c1c546ce5a29163fdad6258645
Jul 01 18:06:54.815734 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.815701    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:06:54.816218 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.815747    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:06:54.816218 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.815710    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:06:54.818422 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.818398    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 18:06:54.818554 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.818426    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pdj6p\""
Jul 01 18:06:54.819776 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.819757    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 18:06:54.819902 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.819783    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 18:06:54.819972 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.819899    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bhlcl\""
Jul 01 18:06:54.819972 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.819945    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 18:06:54.961543 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.961507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:06:54.961707 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.961645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:54.961707 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:54.961681    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 18:06:54.961816 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:54.961753    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert podName:4c69cbc1-40fc-4abc-909b-bd59ced4b2a2 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:56.961732103 +0000 UTC m=+38.529017079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-jm2m4" (UID: "4c69cbc1-40fc-4abc-909b-bd59ced4b2a2") : secret "networking-console-plugin-cert" not found
Jul 01 18:06:54.961816 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:54.961765    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:06:54.961816 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:54.961778    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bdccbb57-zl6tl: secret "image-registry-tls" not found
Jul 01 18:06:54.961961 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:54.961827    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls podName:58c018bb-8a6f-4e8f-8a94-51f9bf03a063 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:56.961812123 +0000 UTC m=+38.529097097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls") pod "image-registry-55bdccbb57-zl6tl" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063") : secret "image-registry-tls" not found
Jul 01 18:06:54.975968 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.975940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk" event={"ID":"dc4e980d-eaf3-4a0b-86b8-c94656918bbd","Type":"ContainerStarted","Data":"4e5f0f028df878dbd4c50764560a9f34b2d7b8c1c546ce5a29163fdad6258645"}
Jul 01 18:06:54.977257 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.977221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dhrff" event={"ID":"fcdd9ed9-9c31-4499-8ace-63a0ec626420","Type":"ContainerStarted","Data":"ad8868d850b458955591d9bbc71503cfc7397aaf84c39dc86546de5c0d18b62b"}
Jul 01 18:06:54.980577 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.980554    2578 generic.go:356] "Generic (PLEG): container finished" podID="298d79e6-cc04-47e5-98a5-36e2284c5174" containerID="e50b35b4096813417196125309193921675d5bd8781db41e0f87c83404353206" exitCode=0
Jul 01 18:06:54.980689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.980624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg6np" event={"ID":"298d79e6-cc04-47e5-98a5-36e2284c5174","Type":"ContainerDied","Data":"e50b35b4096813417196125309193921675d5bd8781db41e0f87c83404353206"}
Jul 01 18:06:54.982068 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.981793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6" event={"ID":"fc0e33ec-c57e-43f2-97cc-28b7a97db026","Type":"ContainerStarted","Data":"382f742f82b23c1aa161425e5e27f61c7a67ac6140affec3ef746d5f2c4a9796"}
Jul 01 18:06:54.983012 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:54.982977    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p" event={"ID":"15516c4a-48cf-4121-9e72-6303618ee567","Type":"ContainerStarted","Data":"8a1fb5bc86a0c5ccceb965601928d5c086a1df0e5cfff60074c93d9a9c2ae6c8"}
Jul 01 18:06:55.062146 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:55.062113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:55.062313 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:55.062294    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:06:55.062378 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:55.062361    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls podName:61840d9c-093c-4b02-8abe-e1db22856cbf nodeName:}" failed. No retries permitted until 2026-07-01 18:06:57.062341361 +0000 UTC m=+38.629626339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls") pod "dns-default-b7ls7" (UID: "61840d9c-093c-4b02-8abe-e1db22856cbf") : secret "dns-default-metrics-tls" not found
Jul 01 18:06:55.990620 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:55.990372    2578 generic.go:356] "Generic (PLEG): container finished" podID="298d79e6-cc04-47e5-98a5-36e2284c5174" containerID="0433d5e8809f2cf7623a4d36ec41faaf6c36b3cfc9c6cd6b8fdcf2d4e373d1cc" exitCode=0
Jul 01 18:06:55.991065 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:55.990558    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg6np" event={"ID":"298d79e6-cc04-47e5-98a5-36e2284c5174","Type":"ContainerDied","Data":"0433d5e8809f2cf7623a4d36ec41faaf6c36b3cfc9c6cd6b8fdcf2d4e373d1cc"}
Jul 01 18:06:56.980771 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:56.980732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:06:56.980926 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:56.980811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:06:56.980926 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:56.980893    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:06:56.980926 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:56.980922    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bdccbb57-zl6tl: secret "image-registry-tls" not found
Jul 01 18:06:56.981092 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:56.980948    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 18:06:56.981092 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:56.980982    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls podName:58c018bb-8a6f-4e8f-8a94-51f9bf03a063 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:00.980964489 +0000 UTC m=+42.548249445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls") pod "image-registry-55bdccbb57-zl6tl" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063") : secret "image-registry-tls" not found
Jul 01 18:06:56.981092 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:56.981028    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert podName:4c69cbc1-40fc-4abc-909b-bd59ced4b2a2 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:00.981008499 +0000 UTC m=+42.548293476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-jm2m4" (UID: "4c69cbc1-40fc-4abc-909b-bd59ced4b2a2") : secret "networking-console-plugin-cert" not found
Jul 01 18:06:57.082253 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:57.082220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:06:57.082710 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:57.082376    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:06:57.082710 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:06:57.082444    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls podName:61840d9c-093c-4b02-8abe-e1db22856cbf nodeName:}" failed. No retries permitted until 2026-07-01 18:07:01.08242445 +0000 UTC m=+42.649709409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls") pod "dns-default-b7ls7" (UID: "61840d9c-093c-4b02-8abe-e1db22856cbf") : secret "dns-default-metrics-tls" not found
Jul 01 18:06:59.002409 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:59.002374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg6np" event={"ID":"298d79e6-cc04-47e5-98a5-36e2284c5174","Type":"ContainerStarted","Data":"c738245ca4360cfe0b5ace19c293cdc66aa25fe3d324705facba917a2539c9e2"}
Jul 01 18:06:59.003961 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:59.003928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk" event={"ID":"dc4e980d-eaf3-4a0b-86b8-c94656918bbd","Type":"ContainerStarted","Data":"5b113d6789464d90a5390e58482c0d48f792eab4d4dc6d1e240d5b760ff06d2c"}
Jul 01 18:06:59.005328 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:59.005303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dhrff" event={"ID":"fcdd9ed9-9c31-4499-8ace-63a0ec626420","Type":"ContainerStarted","Data":"02d4fc548fbf7734cda451b62158754139bcaaee07b91dc023dbce2e0efae1b3"}
Jul 01 18:06:59.025612 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:59.025551    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dg6np" podStartSLOduration=10.185824957 podStartE2EDuration="40.025533475s" totalImagesPullingTime="29.839708518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:59.023958384 +0000 UTC m=+40.591243364" watchObservedRunningTime="2026-07-01 18:06:59.025533475 +0000 UTC m=+40.592818451"
Jul 01 18:06:59.042872 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:59.042816    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dhrff" podStartSLOduration=2.016922035 podStartE2EDuration="6.042797195s" totalImagesPullingTime="4.02587516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:59.042208745 +0000 UTC m=+40.609493745" watchObservedRunningTime="2026-07-01 18:06:59.042797195 +0000 UTC m=+40.610082173"
Jul 01 18:06:59.058101 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:06:59.057986    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk" podStartSLOduration=19.053760553 podStartE2EDuration="23.057965051s" totalImagesPullingTime="4.004204498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:59.057463759 +0000 UTC m=+40.624748753" watchObservedRunningTime="2026-07-01 18:06:59.057965051 +0000 UTC m=+40.625250033"
Jul 01 18:07:01.011983 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:01.011893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6" event={"ID":"fc0e33ec-c57e-43f2-97cc-28b7a97db026","Type":"ContainerStarted","Data":"cad2ffe7a862ee073e8e2e3b2bb1e567af5b87edad764587b245a914e5691716"}
Jul 01 18:07:01.011983 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:01.011939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6" event={"ID":"fc0e33ec-c57e-43f2-97cc-28b7a97db026","Type":"ContainerStarted","Data":"01d62b68b21798c8136a154b79eb6d4b9ab8e47533613ce975b06b3b76327d5f"}
Jul 01 18:07:01.011983 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:01.011954    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6" event={"ID":"fc0e33ec-c57e-43f2-97cc-28b7a97db026","Type":"ContainerStarted","Data":"54791b1e8eeb02132726d39e5100d6e5d8b1d7d753fd7b74b35dfaae8400d316"}
Jul 01 18:07:01.016225 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:01.016203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:07:01.016297 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:01.016247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:07:01.016351 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:01.016333    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:07:01.016384 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:01.016352    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 18:07:01.016413 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:01.016393    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert podName:4c69cbc1-40fc-4abc-909b-bd59ced4b2a2 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:09.016381432 +0000 UTC m=+50.583666390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-jm2m4" (UID: "4c69cbc1-40fc-4abc-909b-bd59ced4b2a2") : secret "networking-console-plugin-cert" not found
Jul 01 18:07:01.016413 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:01.016353    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bdccbb57-zl6tl: secret "image-registry-tls" not found
Jul 01 18:07:01.016485 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:01.016423    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls podName:58c018bb-8a6f-4e8f-8a94-51f9bf03a063 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:09.016416618 +0000 UTC m=+50.583701574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls") pod "image-registry-55bdccbb57-zl6tl" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063") : secret "image-registry-tls" not found
Jul 01 18:07:01.029151 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:01.029104    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6" podStartSLOduration=18.617323121 podStartE2EDuration="25.029088018s" totalImagesPullingTime="6.411764897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:01.028280308 +0000 UTC m=+42.595565289" watchObservedRunningTime="2026-07-01 18:07:01.029088018 +0000 UTC m=+42.596372974"
Jul 01 18:07:01.117298 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:01.117260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:07:01.117567 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:01.117542    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:07:01.117663 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:01.117641    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls podName:61840d9c-093c-4b02-8abe-e1db22856cbf nodeName:}" failed. No retries permitted until 2026-07-01 18:07:09.117614332 +0000 UTC m=+50.684899299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls") pod "dns-default-b7ls7" (UID: "61840d9c-093c-4b02-8abe-e1db22856cbf") : secret "dns-default-metrics-tls" not found
Jul 01 18:07:09.081722 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:09.081689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:07:09.082176 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:09.081741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:07:09.082176 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:09.081886    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:07:09.082176 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:09.081907    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bdccbb57-zl6tl: secret "image-registry-tls" not found
Jul 01 18:07:09.082176 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:09.081954    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls podName:58c018bb-8a6f-4e8f-8a94-51f9bf03a063 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:25.081939947 +0000 UTC m=+66.649224923 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls") pod "image-registry-55bdccbb57-zl6tl" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063") : secret "image-registry-tls" not found
Jul 01 18:07:09.082176 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:09.081892    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 18:07:09.082176 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:09.082039    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert podName:4c69cbc1-40fc-4abc-909b-bd59ced4b2a2 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:25.082025223 +0000 UTC m=+66.649310199 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-jm2m4" (UID: "4c69cbc1-40fc-4abc-909b-bd59ced4b2a2") : secret "networking-console-plugin-cert" not found
Jul 01 18:07:09.182589 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:09.182551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:07:09.182738 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:09.182678    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:07:09.182738 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:09.182736    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls podName:61840d9c-093c-4b02-8abe-e1db22856cbf nodeName:}" failed. No retries permitted until 2026-07-01 18:07:25.182721481 +0000 UTC m=+66.750006438 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls") pod "dns-default-b7ls7" (UID: "61840d9c-093c-4b02-8abe-e1db22856cbf") : secret "dns-default-metrics-tls" not found
Jul 01 18:07:16.049524 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:16.049484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p" event={"ID":"15516c4a-48cf-4121-9e72-6303618ee567","Type":"ContainerStarted","Data":"b8f8a01c275fa4a38343cd1b0e2095f65269ab94e9bf77898abbd1db809c16e1"}
Jul 01 18:07:16.049922 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:16.049674    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:07:16.051345 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:16.051322    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:07:16.080052 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:16.080007    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p" podStartSLOduration=18.819268281 podStartE2EDuration="40.079977772s" totalImagesPullingTime="21.260709491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:16.065132054 +0000 UTC m=+57.632417034" watchObservedRunningTime="2026-07-01 18:07:16.079977772 +0000 UTC m=+57.647262742"
Jul 01 18:07:18.964503 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:18.964478    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldwxn"
Jul 01 18:07:24.706650 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:24.706608    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:07:24.707130 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:24.706660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:07:24.707130 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:24.706685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmqz\" (UniqueName: \"kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz\") pod \"network-check-target-t4vsn\" (UID: \"9ceed3fb-b91c-43c2-8813-7991a3c27659\") " pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:07:24.709465 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:24.709444    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 18:07:24.709522 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:24.709501    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 18:07:24.709703 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:24.709690    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 18:07:24.717740 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:24.717720    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 18:07:24.717821 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:24.717790    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs podName:8936baa9-7b61-4534-b564-a7bb12bf3535 nodeName:}" failed. No retries permitted until 2026-07-01 18:08:28.717768255 +0000 UTC m=+130.285053213 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs") pod "network-metrics-daemon-2r8hd" (UID: "8936baa9-7b61-4534-b564-a7bb12bf3535") : secret "metrics-daemon-secret" not found
Jul 01 18:07:24.719675 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:24.719656    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 18:07:24.721506 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:24.721488    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5f3eafe-b7a2-47e9-954a-028746276935-original-pull-secret\") pod \"global-pull-secret-syncer-q89rm\" (UID: \"c5f3eafe-b7a2-47e9-954a-028746276935\") " pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:07:24.730602 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:24.730580    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmqz\" (UniqueName: \"kubernetes.io/projected/9ceed3fb-b91c-43c2-8813-7991a3c27659-kube-api-access-wvmqz\") pod \"network-check-target-t4vsn\" (UID: \"9ceed3fb-b91c-43c2-8813-7991a3c27659\") " pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:07:24.829224 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:24.829199    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pdj6p\""
Jul 01 18:07:24.837526 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:24.837508    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:07:24.837587 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:24.837516    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q89rm"
Jul 01 18:07:25.003530 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:25.003500    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t4vsn"]
Jul 01 18:07:25.006858 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:07:25.006832    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ceed3fb_b91c_43c2_8813_7991a3c27659.slice/crio-a7bb64e3ce54bdd244a822ab69807fdce8a6a9b591e2fdb6e33e3b4a90879877 WatchSource:0}: Error finding container a7bb64e3ce54bdd244a822ab69807fdce8a6a9b591e2fdb6e33e3b4a90879877: Status 404 returned error can't find the container with id a7bb64e3ce54bdd244a822ab69807fdce8a6a9b591e2fdb6e33e3b4a90879877
Jul 01 18:07:25.019135 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:25.019110    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q89rm"]
Jul 01 18:07:25.022746 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:07:25.022721    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f3eafe_b7a2_47e9_954a_028746276935.slice/crio-0a62443bc323e0781fb1b98b48e6d95aaa0bf804b60b5d3f489b8df5a4618725 WatchSource:0}: Error finding container 0a62443bc323e0781fb1b98b48e6d95aaa0bf804b60b5d3f489b8df5a4618725: Status 404 returned error can't find the container with id 0a62443bc323e0781fb1b98b48e6d95aaa0bf804b60b5d3f489b8df5a4618725
Jul 01 18:07:25.078544 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:25.078515    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q89rm" event={"ID":"c5f3eafe-b7a2-47e9-954a-028746276935","Type":"ContainerStarted","Data":"0a62443bc323e0781fb1b98b48e6d95aaa0bf804b60b5d3f489b8df5a4618725"}
Jul 01 18:07:25.079451 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:25.079433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t4vsn" event={"ID":"9ceed3fb-b91c-43c2-8813-7991a3c27659","Type":"ContainerStarted","Data":"a7bb64e3ce54bdd244a822ab69807fdce8a6a9b591e2fdb6e33e3b4a90879877"}
Jul 01 18:07:25.109837 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:25.109814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:07:25.109938 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:25.109863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:07:25.110017 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:25.109953    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:07:25.110017 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:25.109967    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bdccbb57-zl6tl: secret "image-registry-tls" not found
Jul 01 18:07:25.110017 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:25.109975    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 18:07:25.110165 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:25.110035    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls podName:58c018bb-8a6f-4e8f-8a94-51f9bf03a063 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:57.110016827 +0000 UTC m=+98.677301806 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls") pod "image-registry-55bdccbb57-zl6tl" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063") : secret "image-registry-tls" not found
Jul 01 18:07:25.110165 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:25.110049    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert podName:4c69cbc1-40fc-4abc-909b-bd59ced4b2a2 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:57.110042367 +0000 UTC m=+98.677327323 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-jm2m4" (UID: "4c69cbc1-40fc-4abc-909b-bd59ced4b2a2") : secret "networking-console-plugin-cert" not found
Jul 01 18:07:25.210580 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:25.210510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:07:25.210700 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:25.210651    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:07:25.210764 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:25.210717    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls podName:61840d9c-093c-4b02-8abe-e1db22856cbf nodeName:}" failed. No retries permitted until 2026-07-01 18:07:57.210697417 +0000 UTC m=+98.777982388 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls") pod "dns-default-b7ls7" (UID: "61840d9c-093c-4b02-8abe-e1db22856cbf") : secret "dns-default-metrics-tls" not found
Jul 01 18:07:30.098170 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:30.098128    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t4vsn" event={"ID":"9ceed3fb-b91c-43c2-8813-7991a3c27659","Type":"ContainerStarted","Data":"f0b48fa81e31c0eab16393ea61a4e394d1f432b2509a94d5b190f58084b4ea25"}
Jul 01 18:07:30.098603 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:30.098331    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:07:30.099494 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:30.099470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q89rm" event={"ID":"c5f3eafe-b7a2-47e9-954a-028746276935","Type":"ContainerStarted","Data":"d322ca3eeb49bffb6814d16f6d259e8e6a6593e7e95f3fe7b86d7d149a6fcabc"}
Jul 01 18:07:30.113541 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:30.113498    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t4vsn" podStartSLOduration=66.607604488 podStartE2EDuration="1m11.113484669s" totalImagesPullingTime="4.505880181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:30.112886484 +0000 UTC m=+71.680171464" watchObservedRunningTime="2026-07-01 18:07:30.113484669 +0000 UTC m=+71.680769648"
Jul 01 18:07:30.127832 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:30.127793    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q89rm" podStartSLOduration=65.633713711 podStartE2EDuration="1m10.127782165s" totalImagesPullingTime="4.494068454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:30.126839148 +0000 UTC m=+71.694124148" watchObservedRunningTime="2026-07-01 18:07:30.127782165 +0000 UTC m=+71.695067145"
Jul 01 18:07:57.133437 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:57.133342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:07:57.133437 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:57.133409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:07:57.133872 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:57.133504    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:07:57.133872 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:57.133521    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bdccbb57-zl6tl: secret "image-registry-tls" not found
Jul 01 18:07:57.133872 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:57.133522    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 18:07:57.133872 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:57.133591    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert podName:4c69cbc1-40fc-4abc-909b-bd59ced4b2a2 nodeName:}" failed. No retries permitted until 2026-07-01 18:09:01.133574996 +0000 UTC m=+162.700859953 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-jm2m4" (UID: "4c69cbc1-40fc-4abc-909b-bd59ced4b2a2") : secret "networking-console-plugin-cert" not found
Jul 01 18:07:57.133872 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:57.133606    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls podName:58c018bb-8a6f-4e8f-8a94-51f9bf03a063 nodeName:}" failed. No retries permitted until 2026-07-01 18:09:01.133599509 +0000 UTC m=+162.700884465 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls") pod "image-registry-55bdccbb57-zl6tl" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063") : secret "image-registry-tls" not found
Jul 01 18:07:57.233733 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:07:57.233699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:07:57.233881 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:57.233841    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:07:57.233923 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:07:57.233916    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls podName:61840d9c-093c-4b02-8abe-e1db22856cbf nodeName:}" failed. No retries permitted until 2026-07-01 18:09:01.23390222 +0000 UTC m=+162.801187176 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls") pod "dns-default-b7ls7" (UID: "61840d9c-093c-4b02-8abe-e1db22856cbf") : secret "dns-default-metrics-tls" not found
Jul 01 18:08:01.105456 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:08:01.105424    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t4vsn"
Jul 01 18:08:28.756059 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:08:28.756006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:08:28.756555 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:08:28.756158    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 18:08:28.756555 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:08:28.756236    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs podName:8936baa9-7b61-4534-b564-a7bb12bf3535 nodeName:}" failed. No retries permitted until 2026-07-01 18:10:30.756219687 +0000 UTC m=+252.323504647 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs") pod "network-metrics-daemon-2r8hd" (UID: "8936baa9-7b61-4534-b564-a7bb12bf3535") : secret "metrics-daemon-secret" not found
Jul 01 18:08:56.343544 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:08:56.343499    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl" podUID="58c018bb-8a6f-4e8f-8a94-51f9bf03a063"
Jul 01 18:08:56.357903 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:08:56.357876    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:08:56.380300 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:08:56.380271    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4" podUID="4c69cbc1-40fc-4abc-909b-bd59ced4b2a2"
Jul 01 18:08:56.399636 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:08:56.399610    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-b7ls7" podUID="61840d9c-093c-4b02-8abe-e1db22856cbf"
Jul 01 18:08:57.361953 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:08:57.361915    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:08:57.362427 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:08:57.362405    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:08:57.832804 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:08:57.832774    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2r8hd" podUID="8936baa9-7b61-4534-b564-a7bb12bf3535"
Jul 01 18:08:58.367679 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:08:58.367644    2578 generic.go:356] "Generic (PLEG): container finished" podID="dc4e980d-eaf3-4a0b-86b8-c94656918bbd" containerID="5b113d6789464d90a5390e58482c0d48f792eab4d4dc6d1e240d5b760ff06d2c" exitCode=255
Jul 01 18:08:58.367982 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:08:58.367709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk" event={"ID":"dc4e980d-eaf3-4a0b-86b8-c94656918bbd","Type":"ContainerDied","Data":"5b113d6789464d90a5390e58482c0d48f792eab4d4dc6d1e240d5b760ff06d2c"}
Jul 01 18:08:58.368076 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:08:58.368056    2578 scope.go:117] "RemoveContainer" containerID="5b113d6789464d90a5390e58482c0d48f792eab4d4dc6d1e240d5b760ff06d2c"
Jul 01 18:08:59.373083 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:08:59.373044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk" event={"ID":"dc4e980d-eaf3-4a0b-86b8-c94656918bbd","Type":"ContainerStarted","Data":"02096e4e3d7b9acc3f3dad873f3adfa39250c813581460a9d0e5e3f2693bb8b1"}
Jul 01 18:09:01.188776 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:09:01.188739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:09:01.188776 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:09:01.188786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:09:01.189357 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:09:01.188899    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 18:09:01.189357 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:09:01.188917    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:09:01.189357 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:09:01.188951    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bdccbb57-zl6tl: secret "image-registry-tls" not found
Jul 01 18:09:01.189357 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:09:01.188974    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert podName:4c69cbc1-40fc-4abc-909b-bd59ced4b2a2 nodeName:}" failed. No retries permitted until 2026-07-01 18:11:03.188958278 +0000 UTC m=+284.756243235 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-jm2m4" (UID: "4c69cbc1-40fc-4abc-909b-bd59ced4b2a2") : secret "networking-console-plugin-cert" not found
Jul 01 18:09:01.189357 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:09:01.189038    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls podName:58c018bb-8a6f-4e8f-8a94-51f9bf03a063 nodeName:}" failed. No retries permitted until 2026-07-01 18:11:03.189015816 +0000 UTC m=+284.756300833 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls") pod "image-registry-55bdccbb57-zl6tl" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063") : secret "image-registry-tls" not found
Jul 01 18:09:01.289458 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:09:01.289424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:09:01.289602 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:09:01.289565    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:09:01.289658 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:09:01.289626    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls podName:61840d9c-093c-4b02-8abe-e1db22856cbf nodeName:}" failed. No retries permitted until 2026-07-01 18:11:03.289612728 +0000 UTC m=+284.856897686 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls") pod "dns-default-b7ls7" (UID: "61840d9c-093c-4b02-8abe-e1db22856cbf") : secret "dns-default-metrics-tls" not found
Jul 01 18:09:10.815961 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:09:10.815915    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:09:16.050739 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:09:16.050689    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p" podUID="15516c4a-48cf-4121-9e72-6303618ee567" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.9:8000/readyz\": dial tcp 10.132.0.9:8000: connect: connection refused"
Jul 01 18:09:16.427187 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:09:16.427101    2578 generic.go:356] "Generic (PLEG): container finished" podID="15516c4a-48cf-4121-9e72-6303618ee567" containerID="b8f8a01c275fa4a38343cd1b0e2095f65269ab94e9bf77898abbd1db809c16e1" exitCode=1
Jul 01 18:09:16.427187 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:09:16.427162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p" event={"ID":"15516c4a-48cf-4121-9e72-6303618ee567","Type":"ContainerDied","Data":"b8f8a01c275fa4a38343cd1b0e2095f65269ab94e9bf77898abbd1db809c16e1"}
Jul 01 18:09:16.427470 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:09:16.427456    2578 scope.go:117] "RemoveContainer" containerID="b8f8a01c275fa4a38343cd1b0e2095f65269ab94e9bf77898abbd1db809c16e1"
Jul 01 18:09:17.435261 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:09:17.435217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p" event={"ID":"15516c4a-48cf-4121-9e72-6303618ee567","Type":"ContainerStarted","Data":"a6d6456df7146ffa57ba5a36f11cc739eb3b2aef381295cf814c745ad6de8a88"}
Jul 01 18:09:17.435772 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:09:17.435622    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:09:17.436286 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:09:17.436268    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:10:09.829524 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.829486    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zxvp7"]
Jul 01 18:10:09.832689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.832665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:09.835453 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.835427    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 18:10:09.835585 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.835468    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9p988\""
Jul 01 18:10:09.835585 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.835430    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 18:10:09.835585 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.835428    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 18:10:09.835585 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.835575    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 18:10:09.844682 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.844652    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zxvp7"]
Jul 01 18:10:09.883716 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.883678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-crio-socket\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:09.883716 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.883721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:09.883944 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.883747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps4zs\" (UniqueName: \"kubernetes.io/projected/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-api-access-ps4zs\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:09.883944 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.883782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-data-volume\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:09.883944 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.883887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:09.984443 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.984399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-data-volume\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:09.984595 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.984507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:09.984595 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.984532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-crio-socket\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:09.984595 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.984553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:09.984595 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.984578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps4zs\" (UniqueName: \"kubernetes.io/projected/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-api-access-ps4zs\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:09.984721 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.984645    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-crio-socket\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:09.984721 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:09.984663    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:10:09.984782 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:09.984663    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:10:09.984782 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:09.984750    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls podName:c7cc4faf-c31f-4cf1-8461-61b5e3827dba nodeName:}" failed. No retries permitted until 2026-07-01 18:10:10.484728456 +0000 UTC m=+232.052013431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zxvp7" (UID: "c7cc4faf-c31f-4cf1-8461-61b5e3827dba") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:10:09.984782 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.984755    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-data-volume\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:09.984898 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:09.984785    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm podName:c7cc4faf-c31f-4cf1-8461-61b5e3827dba nodeName:}" failed. No retries permitted until 2026-07-01 18:10:10.484772209 +0000 UTC m=+232.052057166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zxvp7" (UID: "c7cc4faf-c31f-4cf1-8461-61b5e3827dba") : configmap "kube-rbac-proxy" not found
Jul 01 18:10:09.993102 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:09.993076    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps4zs\" (UniqueName: \"kubernetes.io/projected/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-api-access-ps4zs\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:10.489665 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:10.489626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:10.489665 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:10.489674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:10.489892 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:10.489740    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:10:10.489892 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:10.489780    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:10:10.489892 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:10.489793    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm podName:c7cc4faf-c31f-4cf1-8461-61b5e3827dba nodeName:}" failed. No retries permitted until 2026-07-01 18:10:11.489778836 +0000 UTC m=+233.057063793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zxvp7" (UID: "c7cc4faf-c31f-4cf1-8461-61b5e3827dba") : configmap "kube-rbac-proxy" not found
Jul 01 18:10:10.489892 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:10.489867    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls podName:c7cc4faf-c31f-4cf1-8461-61b5e3827dba nodeName:}" failed. No retries permitted until 2026-07-01 18:10:11.489848023 +0000 UTC m=+233.057132985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zxvp7" (UID: "c7cc4faf-c31f-4cf1-8461-61b5e3827dba") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:10:11.497306 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:11.497251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:11.497306 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:11.497314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:11.497722 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:11.497378    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:10:11.497722 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:11.497406    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:10:11.497722 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:11.497461    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm podName:c7cc4faf-c31f-4cf1-8461-61b5e3827dba nodeName:}" failed. No retries permitted until 2026-07-01 18:10:13.497445773 +0000 UTC m=+235.064730730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zxvp7" (UID: "c7cc4faf-c31f-4cf1-8461-61b5e3827dba") : configmap "kube-rbac-proxy" not found
Jul 01 18:10:11.497722 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:11.497474    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls podName:c7cc4faf-c31f-4cf1-8461-61b5e3827dba nodeName:}" failed. No retries permitted until 2026-07-01 18:10:13.497467927 +0000 UTC m=+235.064752884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zxvp7" (UID: "c7cc4faf-c31f-4cf1-8461-61b5e3827dba") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:10:12.034722 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:12.034690    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hmcsv_4ce225ff-82c4-49f3-9dd0-b8df3d1b196d/dns-node-resolver/0.log"
Jul 01 18:10:12.832385 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:12.832359    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bbppr_9d1e4c3c-8594-4a71-86fe-9e77d4f4153f/node-ca/0.log"
Jul 01 18:10:13.513675 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:13.513631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:13.513893 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:13.513767    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:10:13.513893 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:13.513834    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm podName:c7cc4faf-c31f-4cf1-8461-61b5e3827dba nodeName:}" failed. No retries permitted until 2026-07-01 18:10:17.513814515 +0000 UTC m=+239.081099472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zxvp7" (UID: "c7cc4faf-c31f-4cf1-8461-61b5e3827dba") : configmap "kube-rbac-proxy" not found
Jul 01 18:10:13.513893 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:13.513768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:13.513893 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:13.513855    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:10:13.514113 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:13.513907    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls podName:c7cc4faf-c31f-4cf1-8461-61b5e3827dba nodeName:}" failed. No retries permitted until 2026-07-01 18:10:17.513891338 +0000 UTC m=+239.081176295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zxvp7" (UID: "c7cc4faf-c31f-4cf1-8461-61b5e3827dba") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:10:13.630647 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:13.630603    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dhrff_fcdd9ed9-9c31-4499-8ace-63a0ec626420/serve-healthcheck-canary/0.log"
Jul 01 18:10:17.548048 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:17.547980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:17.548514 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:17.548112    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:10:17.548514 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:17.548148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:17.548514 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:17.548181    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm podName:c7cc4faf-c31f-4cf1-8461-61b5e3827dba nodeName:}" failed. No retries permitted until 2026-07-01 18:10:25.548162695 +0000 UTC m=+247.115447664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zxvp7" (UID: "c7cc4faf-c31f-4cf1-8461-61b5e3827dba") : configmap "kube-rbac-proxy" not found
Jul 01 18:10:17.548514 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:17.548231    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:10:17.548514 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:17.548266    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls podName:c7cc4faf-c31f-4cf1-8461-61b5e3827dba nodeName:}" failed. No retries permitted until 2026-07-01 18:10:25.548257151 +0000 UTC m=+247.115542107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zxvp7" (UID: "c7cc4faf-c31f-4cf1-8461-61b5e3827dba") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:10:25.615252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:25.615210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:25.615640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:25.615264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:25.615760 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:25.615743    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:25.617664 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:25.617639    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7cc4faf-c31f-4cf1-8461-61b5e3827dba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zxvp7\" (UID: \"c7cc4faf-c31f-4cf1-8461-61b5e3827dba\") " pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:25.744544 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:25.744505    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9p988\""
Jul 01 18:10:25.752666 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:25.752631    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zxvp7"
Jul 01 18:10:25.881360 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:25.881283    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zxvp7"]
Jul 01 18:10:25.884665 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:10:25.884632    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cc4faf_c31f_4cf1_8461_61b5e3827dba.slice/crio-2e153c9e08a631e9ee56cc15a75a22408b7163b4f00bd3bfa06697fb31fb6445 WatchSource:0}: Error finding container 2e153c9e08a631e9ee56cc15a75a22408b7163b4f00bd3bfa06697fb31fb6445: Status 404 returned error can't find the container with id 2e153c9e08a631e9ee56cc15a75a22408b7163b4f00bd3bfa06697fb31fb6445
Jul 01 18:10:26.648438 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:26.648391    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zxvp7" event={"ID":"c7cc4faf-c31f-4cf1-8461-61b5e3827dba","Type":"ContainerStarted","Data":"2e153c9e08a631e9ee56cc15a75a22408b7163b4f00bd3bfa06697fb31fb6445"}
Jul 01 18:10:29.659785 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:29.659755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zxvp7" event={"ID":"c7cc4faf-c31f-4cf1-8461-61b5e3827dba","Type":"ContainerStarted","Data":"9bd5fabe0d733093b4ace52e0c4096d21b99ea9437139a3c6745414246b01938"}
Jul 01 18:10:30.665042 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:30.664981    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zxvp7" event={"ID":"c7cc4faf-c31f-4cf1-8461-61b5e3827dba","Type":"ContainerStarted","Data":"fb30009da2bd237918f1f965c6663804d4bd8fdec497f3c8e43d47181fe565cc"}
Jul 01 18:10:30.856565 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:30.856519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:10:30.859304 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:30.859272    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8936baa9-7b61-4534-b564-a7bb12bf3535-metrics-certs\") pod \"network-metrics-daemon-2r8hd\" (UID: \"8936baa9-7b61-4534-b564-a7bb12bf3535\") " pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:10:30.919595 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:30.919514    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bhlcl\""
Jul 01 18:10:30.927723 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:30.927689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r8hd"
Jul 01 18:10:31.095461 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:31.095346    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2r8hd"]
Jul 01 18:10:31.098562 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:10:31.098526    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8936baa9_7b61_4534_b564_a7bb12bf3535.slice/crio-fe9a6ee8ec0e66fb57ebe7d287bea516ee4de0a27abfe05a1b5cdc8536405955 WatchSource:0}: Error finding container fe9a6ee8ec0e66fb57ebe7d287bea516ee4de0a27abfe05a1b5cdc8536405955: Status 404 returned error can't find the container with id fe9a6ee8ec0e66fb57ebe7d287bea516ee4de0a27abfe05a1b5cdc8536405955
Jul 01 18:10:31.670032 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:31.669973    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2r8hd" event={"ID":"8936baa9-7b61-4534-b564-a7bb12bf3535","Type":"ContainerStarted","Data":"fe9a6ee8ec0e66fb57ebe7d287bea516ee4de0a27abfe05a1b5cdc8536405955"}
Jul 01 18:10:33.681895 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:33.678292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zxvp7" event={"ID":"c7cc4faf-c31f-4cf1-8461-61b5e3827dba","Type":"ContainerStarted","Data":"06c70472ee128dab3c669ed2aadee49db674d3dd286a38192b7bd9cf29fd0f47"}
Jul 01 18:10:33.684177 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:33.684150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2r8hd" event={"ID":"8936baa9-7b61-4534-b564-a7bb12bf3535","Type":"ContainerStarted","Data":"6c8e4787d8b3dd951fe5d6dae329164b7a52f2317497bc9da4f871cc8371a0af"}
Jul 01 18:10:33.684326 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:33.684197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2r8hd" event={"ID":"8936baa9-7b61-4534-b564-a7bb12bf3535","Type":"ContainerStarted","Data":"d97bbbcb19851d4ae40e5156da17c95eb4e84051ab221d68a9c1cebaa6ef4c93"}
Jul 01 18:10:33.698136 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:33.698085    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zxvp7" podStartSLOduration=17.993388935 podStartE2EDuration="24.698066954s" totalImagesPullingTime="6.704678019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:10:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:10:33.697498544 +0000 UTC m=+255.264783562" watchObservedRunningTime="2026-07-01 18:10:33.698066954 +0000 UTC m=+255.265351936"
Jul 01 18:10:33.713318 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:33.713262    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2r8hd" podStartSLOduration=252.561097547 podStartE2EDuration="4m14.71324388s" totalImagesPullingTime="2.152146333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:10:33.7127706 +0000 UTC m=+255.280055578" watchObservedRunningTime="2026-07-01 18:10:33.71324388 +0000 UTC m=+255.280528918"
Jul 01 18:10:46.914363 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.914280    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5j2nn"]
Jul 01 18:10:46.917463 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.917439    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:46.920210 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.920189    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 18:10:46.920374 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.920358    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 18:10:46.920802 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.920760    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 18:10:46.921453 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.921435    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-js6g5\""
Jul 01 18:10:46.921551 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.921461    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 18:10:46.921551 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.921488    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 18:10:46.921551 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.921495    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 18:10:46.976262 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.976222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-wtmp\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:46.976435 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.976269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8bfa651-2090-43fc-9bf2-33c9314742b9-root\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:46.976435 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.976380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-textfile\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:46.976435 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.976413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:46.976566 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.976444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z885r\" (UniqueName: \"kubernetes.io/projected/a8bfa651-2090-43fc-9bf2-33c9314742b9-kube-api-access-z885r\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:46.976566 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.976471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8bfa651-2090-43fc-9bf2-33c9314742b9-sys\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:46.976566 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.976487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-tls\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:46.976566 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.976509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8bfa651-2090-43fc-9bf2-33c9314742b9-metrics-client-ca\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:46.976566 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:46.976537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.077265 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.077221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-wtmp\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.077265 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.077271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8bfa651-2090-43fc-9bf2-33c9314742b9-root\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.077509 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.077335    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8bfa651-2090-43fc-9bf2-33c9314742b9-root\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.077509 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.077387    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-wtmp\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.077509 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.077397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-textfile\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.077509 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.077428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.077509 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.077463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z885r\" (UniqueName: \"kubernetes.io/projected/a8bfa651-2090-43fc-9bf2-33c9314742b9-kube-api-access-z885r\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.077509 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.077482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8bfa651-2090-43fc-9bf2-33c9314742b9-sys\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.077509 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.077500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-tls\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.077847 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.077549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8bfa651-2090-43fc-9bf2-33c9314742b9-metrics-client-ca\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.077847 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.077572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.077847 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.077548    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8bfa651-2090-43fc-9bf2-33c9314742b9-sys\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.077847 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.077753    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-textfile\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.078151 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.078125    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8bfa651-2090-43fc-9bf2-33c9314742b9-metrics-client-ca\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.078243 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.078144    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.079780 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.079759    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.079947 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.079928    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8bfa651-2090-43fc-9bf2-33c9314742b9-node-exporter-tls\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.086952 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.086932    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z885r\" (UniqueName: \"kubernetes.io/projected/a8bfa651-2090-43fc-9bf2-33c9314742b9-kube-api-access-z885r\") pod \"node-exporter-5j2nn\" (UID: \"a8bfa651-2090-43fc-9bf2-33c9314742b9\") " pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.227365 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.227269    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5j2nn"
Jul 01 18:10:47.238286 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:10:47.238258    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bfa651_2090_43fc_9bf2_33c9314742b9.slice/crio-87a994cfc00427d70a36c861ae19ca7adb3f84922b136c48fac4bc13472b261c WatchSource:0}: Error finding container 87a994cfc00427d70a36c861ae19ca7adb3f84922b136c48fac4bc13472b261c: Status 404 returned error can't find the container with id 87a994cfc00427d70a36c861ae19ca7adb3f84922b136c48fac4bc13472b261c
Jul 01 18:10:47.727733 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:47.727694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5j2nn" event={"ID":"a8bfa651-2090-43fc-9bf2-33c9314742b9","Type":"ContainerStarted","Data":"87a994cfc00427d70a36c861ae19ca7adb3f84922b136c48fac4bc13472b261c"}
Jul 01 18:10:49.737091 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:49.737055    2578 generic.go:356] "Generic (PLEG): container finished" podID="a8bfa651-2090-43fc-9bf2-33c9314742b9" containerID="41722bc4a1f98d00b30d3f4b607247b1e555d828a71ba8f1de6a86d884c2f37f" exitCode=0
Jul 01 18:10:49.737495 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:49.737124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5j2nn" event={"ID":"a8bfa651-2090-43fc-9bf2-33c9314742b9","Type":"ContainerDied","Data":"41722bc4a1f98d00b30d3f4b607247b1e555d828a71ba8f1de6a86d884c2f37f"}
Jul 01 18:10:50.743542 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:50.743511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5j2nn" event={"ID":"a8bfa651-2090-43fc-9bf2-33c9314742b9","Type":"ContainerStarted","Data":"25f79da5f688755310e461c63ffcf91931c07e859af70d3bc6efe785334cae96"}
Jul 01 18:10:50.744038 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:50.743563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5j2nn" event={"ID":"a8bfa651-2090-43fc-9bf2-33c9314742b9","Type":"ContainerStarted","Data":"9c50176a8583b5bc5cafbb3a0294d00ba72a8d6dd20349de4d8910028a0a2435"}
Jul 01 18:10:50.766550 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:50.766498    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5j2nn" podStartSLOduration=3.038717481 podStartE2EDuration="4.766480728s" totalImagesPullingTime="1.662594389s" totalInitContainerRuntime="65.168858ms" isStatefulPod=true podCreationTimestamp="2026-07-01 18:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:10:50.76460414 +0000 UTC m=+272.331889118" watchObservedRunningTime="2026-07-01 18:10:50.766480728 +0000 UTC m=+272.333765740"
Jul 01 18:10:53.660389 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:53.660348    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6" podUID="fc0e33ec-c57e-43f2-97cc-28b7a97db026" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 01 18:10:58.770511 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:58.770429    2578 generic.go:356] "Generic (PLEG): container finished" podID="dc4e980d-eaf3-4a0b-86b8-c94656918bbd" containerID="02096e4e3d7b9acc3f3dad873f3adfa39250c813581460a9d0e5e3f2693bb8b1" exitCode=255
Jul 01 18:10:58.770511 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:58.770485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk" event={"ID":"dc4e980d-eaf3-4a0b-86b8-c94656918bbd","Type":"ContainerDied","Data":"02096e4e3d7b9acc3f3dad873f3adfa39250c813581460a9d0e5e3f2693bb8b1"}
Jul 01 18:10:58.770924 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:58.770532    2578 scope.go:117] "RemoveContainer" containerID="5b113d6789464d90a5390e58482c0d48f792eab4d4dc6d1e240d5b760ff06d2c"
Jul 01 18:10:58.770924 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:58.770871    2578 scope.go:117] "RemoveContainer" containerID="02096e4e3d7b9acc3f3dad873f3adfa39250c813581460a9d0e5e3f2693bb8b1"
Jul 01 18:10:58.771076 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:58.771054    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-768c67f45d-2mbtk_open-cluster-management-agent-addon(dc4e980d-eaf3-4a0b-86b8-c94656918bbd)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk" podUID="dc4e980d-eaf3-4a0b-86b8-c94656918bbd"
Jul 01 18:10:59.358854 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:10:59.358811    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl" podUID="58c018bb-8a6f-4e8f-8a94-51f9bf03a063"
Jul 01 18:10:59.774833 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:10:59.774756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:11:00.363370 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:11:00.363318    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4" podUID="4c69cbc1-40fc-4abc-909b-bd59ced4b2a2"
Jul 01 18:11:00.363370 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:11:00.363342    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-b7ls7" podUID="61840d9c-093c-4b02-8abe-e1db22856cbf"
Jul 01 18:11:00.778110 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:00.778087    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:11:00.778531 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:00.778121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:11:03.201579 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.201540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:11:03.201957 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.201592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:11:03.204677 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.204651    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c69cbc1-40fc-4abc-909b-bd59ced4b2a2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-jm2m4\" (UID: \"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:11:03.204750 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.204651    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls\") pod \"image-registry-55bdccbb57-zl6tl\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") " pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:11:03.302087 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.302051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:11:03.304435 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.304409    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61840d9c-093c-4b02-8abe-e1db22856cbf-metrics-tls\") pod \"dns-default-b7ls7\" (UID: \"61840d9c-093c-4b02-8abe-e1db22856cbf\") " pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:11:03.378028 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.377978    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-86d4l\""
Jul 01 18:11:03.386264 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.386215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:11:03.480871 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.480783    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fnvvx\""
Jul 01 18:11:03.480871 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.480828    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vsbfc\""
Jul 01 18:11:03.488772 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.488737    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"
Jul 01 18:11:03.488937 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.488828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:11:03.661249 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.661200    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6" podUID="fc0e33ec-c57e-43f2-97cc-28b7a97db026" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 01 18:11:03.668200 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.668154    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk"
Jul 01 18:11:03.668618 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.668598    2578 scope.go:117] "RemoveContainer" containerID="02096e4e3d7b9acc3f3dad873f3adfa39250c813581460a9d0e5e3f2693bb8b1"
Jul 01 18:11:03.668842 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:11:03.668817    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-768c67f45d-2mbtk_open-cluster-management-agent-addon(dc4e980d-eaf3-4a0b-86b8-c94656918bbd)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk" podUID="dc4e980d-eaf3-4a0b-86b8-c94656918bbd"
Jul 01 18:11:03.716496 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.716464    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55bdccbb57-zl6tl"]
Jul 01 18:11:03.720491 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:11:03.720460    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c018bb_8a6f_4e8f_8a94_51f9bf03a063.slice/crio-b6385c5e2b98dd1701f86138c97405433edca63db88dc8861f14ceac461ab2dc WatchSource:0}: Error finding container b6385c5e2b98dd1701f86138c97405433edca63db88dc8861f14ceac461ab2dc: Status 404 returned error can't find the container with id b6385c5e2b98dd1701f86138c97405433edca63db88dc8861f14ceac461ab2dc
Jul 01 18:11:03.730173 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.730145    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4"]
Jul 01 18:11:03.734300 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:11:03.734267    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c69cbc1_40fc_4abc_909b_bd59ced4b2a2.slice/crio-e840f603b465fca44ed406f4bcb0268c6912384464c6b3f4b2ec6840e779a0b7 WatchSource:0}: Error finding container e840f603b465fca44ed406f4bcb0268c6912384464c6b3f4b2ec6840e779a0b7: Status 404 returned error can't find the container with id e840f603b465fca44ed406f4bcb0268c6912384464c6b3f4b2ec6840e779a0b7
Jul 01 18:11:03.765395 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.765363    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b7ls7"]
Jul 01 18:11:03.781246 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:11:03.781211    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61840d9c_093c_4b02_8abe_e1db22856cbf.slice/crio-1256de04785cf1f015f75635b3e236411224b9be9c2fa0adbaca190ea97fee72 WatchSource:0}: Error finding container 1256de04785cf1f015f75635b3e236411224b9be9c2fa0adbaca190ea97fee72: Status 404 returned error can't find the container with id 1256de04785cf1f015f75635b3e236411224b9be9c2fa0adbaca190ea97fee72
Jul 01 18:11:03.791198 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.791170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b7ls7" event={"ID":"61840d9c-093c-4b02-8abe-e1db22856cbf","Type":"ContainerStarted","Data":"1256de04785cf1f015f75635b3e236411224b9be9c2fa0adbaca190ea97fee72"}
Jul 01 18:11:03.792315 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.792292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4" event={"ID":"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2","Type":"ContainerStarted","Data":"e840f603b465fca44ed406f4bcb0268c6912384464c6b3f4b2ec6840e779a0b7"}
Jul 01 18:11:03.793393 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:03.793373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl" event={"ID":"58c018bb-8a6f-4e8f-8a94-51f9bf03a063","Type":"ContainerStarted","Data":"b6385c5e2b98dd1701f86138c97405433edca63db88dc8861f14ceac461ab2dc"}
Jul 01 18:11:04.800119 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:04.799693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl" event={"ID":"58c018bb-8a6f-4e8f-8a94-51f9bf03a063","Type":"ContainerStarted","Data":"85f0087485bb7feeaf7bcd647d7ea80dda11cc033e0bb8cdf3e9b239f342352b"}
Jul 01 18:11:04.800119 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:04.799812    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:11:04.824279 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:04.824223    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl" podStartSLOduration=284.824204438 podStartE2EDuration="4m44.824204438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:11:04.821914104 +0000 UTC m=+286.389199084" watchObservedRunningTime="2026-07-01 18:11:04.824204438 +0000 UTC m=+286.391489418"
Jul 01 18:11:05.805690 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:05.805647    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4" event={"ID":"4c69cbc1-40fc-4abc-909b-bd59ced4b2a2","Type":"ContainerStarted","Data":"b0e4572f7760dc7693e5656ec321b7bd85ebb5ded0618151e7c8a905f1bcabf2"}
Jul 01 18:11:05.821846 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:05.821781    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-jm2m4" podStartSLOduration=284.203871361 podStartE2EDuration="4m45.821756039s" totalImagesPullingTime="1.617884678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:11:05.820699136 +0000 UTC m=+287.387984478" watchObservedRunningTime="2026-07-01 18:11:05.821756039 +0000 UTC m=+287.389041019"
Jul 01 18:11:06.810880 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:06.810845    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b7ls7" event={"ID":"61840d9c-093c-4b02-8abe-e1db22856cbf","Type":"ContainerStarted","Data":"6a0cd6ac9a264192be075928c4f4d924c05e851ab7c169e38b3de94db98dc6e6"}
Jul 01 18:11:06.811284 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:06.810894    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b7ls7" event={"ID":"61840d9c-093c-4b02-8abe-e1db22856cbf","Type":"ContainerStarted","Data":"cd78755c0a2c4d0002e0fc1cc970f394cb1bc91bb36e3cc9b1418239ddfd7813"}
Jul 01 18:11:06.828260 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:06.828213    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-b7ls7" podStartSLOduration=251.312374578 podStartE2EDuration="4m13.82819853s" totalImagesPullingTime="2.515823952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:11:06.826695229 +0000 UTC m=+288.393980208" watchObservedRunningTime="2026-07-01 18:11:06.82819853 +0000 UTC m=+288.395483508"
Jul 01 18:11:07.814104 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:07.814074    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:11:13.660745 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:13.660704    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6" podUID="fc0e33ec-c57e-43f2-97cc-28b7a97db026" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 01 18:11:13.661145 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:13.660775    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6"
Jul 01 18:11:13.661302 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:13.661259    2578 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"cad2ffe7a862ee073e8e2e3b2bb1e567af5b87edad764587b245a914e5691716"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 01 18:11:13.661337 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:13.661323    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6" podUID="fc0e33ec-c57e-43f2-97cc-28b7a97db026" containerName="service-proxy" containerID="cri-o://cad2ffe7a862ee073e8e2e3b2bb1e567af5b87edad764587b245a914e5691716" gracePeriod=30
Jul 01 18:11:13.835013 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:13.834955    2578 generic.go:356] "Generic (PLEG): container finished" podID="fc0e33ec-c57e-43f2-97cc-28b7a97db026" containerID="cad2ffe7a862ee073e8e2e3b2bb1e567af5b87edad764587b245a914e5691716" exitCode=2
Jul 01 18:11:13.835169 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:13.835029    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6" event={"ID":"fc0e33ec-c57e-43f2-97cc-28b7a97db026","Type":"ContainerDied","Data":"cad2ffe7a862ee073e8e2e3b2bb1e567af5b87edad764587b245a914e5691716"}
Jul 01 18:11:13.835169 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:13.835084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-774584c764-kkjr6" event={"ID":"fc0e33ec-c57e-43f2-97cc-28b7a97db026","Type":"ContainerStarted","Data":"17ba7874b03e58185ea0052ad87a5a4d1144a08270bb4401cd1e9c5d65e5bd73"}
Jul 01 18:11:16.816426 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:16.816388    2578 scope.go:117] "RemoveContainer" containerID="02096e4e3d7b9acc3f3dad873f3adfa39250c813581460a9d0e5e3f2693bb8b1"
Jul 01 18:11:16.847107 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:16.847072    2578 generic.go:356] "Generic (PLEG): container finished" podID="15516c4a-48cf-4121-9e72-6303618ee567" containerID="a6d6456df7146ffa57ba5a36f11cc739eb3b2aef381295cf814c745ad6de8a88" exitCode=1
Jul 01 18:11:16.847262 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:16.847151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p" event={"ID":"15516c4a-48cf-4121-9e72-6303618ee567","Type":"ContainerDied","Data":"a6d6456df7146ffa57ba5a36f11cc739eb3b2aef381295cf814c745ad6de8a88"}
Jul 01 18:11:16.847262 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:16.847213    2578 scope.go:117] "RemoveContainer" containerID="b8f8a01c275fa4a38343cd1b0e2095f65269ab94e9bf77898abbd1db809c16e1"
Jul 01 18:11:16.847519 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:16.847504    2578 scope.go:117] "RemoveContainer" containerID="a6d6456df7146ffa57ba5a36f11cc739eb3b2aef381295cf814c745ad6de8a88"
Jul 01 18:11:16.847726 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:11:16.847703    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-594b6c4d4d-fmr7p_open-cluster-management-agent-addon(15516c4a-48cf-4121-9e72-6303618ee567)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p" podUID="15516c4a-48cf-4121-9e72-6303618ee567"
Jul 01 18:11:17.436520 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:17.436469    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:11:17.819959 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:17.819928    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b7ls7"
Jul 01 18:11:17.853315 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:17.853284    2578 scope.go:117] "RemoveContainer" containerID="a6d6456df7146ffa57ba5a36f11cc739eb3b2aef381295cf814c745ad6de8a88"
Jul 01 18:11:17.853498 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:11:17.853468    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-594b6c4d4d-fmr7p_open-cluster-management-agent-addon(15516c4a-48cf-4121-9e72-6303618ee567)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p" podUID="15516c4a-48cf-4121-9e72-6303618ee567"
Jul 01 18:11:17.854961 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:17.854936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-768c67f45d-2mbtk" event={"ID":"dc4e980d-eaf3-4a0b-86b8-c94656918bbd","Type":"ContainerStarted","Data":"63e16725527aa35803009f39c7edfa4f47c1eb6584a8ff064c65fb9f349a5b00"}
Jul 01 18:11:18.761470 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:18.761438    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:11:18.762828 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:18.762805    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:11:18.768445 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:18.768422    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 18:11:23.390913 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:23.390874    2578 patch_prober.go:28] interesting pod/image-registry-55bdccbb57-zl6tl container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 01 18:11:23.395609 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:23.390938    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl" podUID="58c018bb-8a6f-4e8f-8a94-51f9bf03a063" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:11:23.689355 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:23.689256    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:11:23.689699 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:23.689686    2578 scope.go:117] "RemoveContainer" containerID="a6d6456df7146ffa57ba5a36f11cc739eb3b2aef381295cf814c745ad6de8a88"
Jul 01 18:11:23.689851 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:11:23.689833    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-594b6c4d4d-fmr7p_open-cluster-management-agent-addon(15516c4a-48cf-4121-9e72-6303618ee567)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p" podUID="15516c4a-48cf-4121-9e72-6303618ee567"
Jul 01 18:11:25.025987 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:25.025951    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5j2nn_a8bfa651-2090-43fc-9bf2-33c9314742b9/init-textfile/0.log"
Jul 01 18:11:25.227273 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:25.227244    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5j2nn_a8bfa651-2090-43fc-9bf2-33c9314742b9/node-exporter/0.log"
Jul 01 18:11:25.425420 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:25.425338    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5j2nn_a8bfa651-2090-43fc-9bf2-33c9314742b9/kube-rbac-proxy/0.log"
Jul 01 18:11:25.810005 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:25.809962    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:11:30.625888 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:30.625854    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-jm2m4_4c69cbc1-40fc-4abc-909b-bd59ced4b2a2/networking-console-plugin/0.log"
Jul 01 18:11:31.827128 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:31.827093    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b7ls7_61840d9c-093c-4b02-8abe-e1db22856cbf/dns/0.log"
Jul 01 18:11:32.028762 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:32.028727    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b7ls7_61840d9c-093c-4b02-8abe-e1db22856cbf/kube-rbac-proxy/0.log"
Jul 01 18:11:33.219034 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:33.218978    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55bdccbb57-zl6tl"]
Jul 01 18:11:33.425275 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:33.425247    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hmcsv_4ce225ff-82c4-49f3-9dd0-b8df3d1b196d/dns-node-resolver/0.log"
Jul 01 18:11:33.625704 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:33.625674    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55bdccbb57-zl6tl_58c018bb-8a6f-4e8f-8a94-51f9bf03a063/registry/0.log"
Jul 01 18:11:34.225308 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:34.225283    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bbppr_9d1e4c3c-8594-4a71-86fe-9e77d4f4153f/node-ca/0.log"
Jul 01 18:11:35.025813 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:35.025786    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dhrff_fcdd9ed9-9c31-4499-8ace-63a0ec626420/serve-healthcheck-canary/0.log"
Jul 01 18:11:35.816062 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:35.816025    2578 scope.go:117] "RemoveContainer" containerID="a6d6456df7146ffa57ba5a36f11cc739eb3b2aef381295cf814c745ad6de8a88"
Jul 01 18:11:35.919505 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:35.919464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p" event={"ID":"15516c4a-48cf-4121-9e72-6303618ee567","Type":"ContainerStarted","Data":"b37bbfd2a670a205583add552bcf5ec10cd0051dc3e81475e088875a4850aff4"}
Jul 01 18:11:35.919795 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:35.919767    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:11:35.921527 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:35.921506    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594b6c4d4d-fmr7p"
Jul 01 18:11:58.239323 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:58.239254    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl" podUID="58c018bb-8a6f-4e8f-8a94-51f9bf03a063" containerName="registry" containerID="cri-o://85f0087485bb7feeaf7bcd647d7ea80dda11cc033e0bb8cdf3e9b239f342352b" gracePeriod=30
Jul 01 18:11:59.482247 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.482216    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:11:59.622503 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.622394    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-installation-pull-secrets\") pod \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") "
Jul 01 18:11:59.622503 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.622444    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdfwm\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-kube-api-access-jdfwm\") pod \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") "
Jul 01 18:11:59.622503 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.622470    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-ca-trust-extracted\") pod \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") "
Jul 01 18:11:59.622790 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.622577    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-trusted-ca\") pod \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") "
Jul 01 18:11:59.622790 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.622612    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-bound-sa-token\") pod \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") "
Jul 01 18:11:59.622790 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.622645    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-certificates\") pod \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") "
Jul 01 18:11:59.622790 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.622686    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-image-registry-private-configuration\") pod \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") "
Jul 01 18:11:59.622790 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.622730    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls\") pod \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\" (UID: \"58c018bb-8a6f-4e8f-8a94-51f9bf03a063\") "
Jul 01 18:11:59.623096 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.623061    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "58c018bb-8a6f-4e8f-8a94-51f9bf03a063" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:11:59.623715 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.623683    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "58c018bb-8a6f-4e8f-8a94-51f9bf03a063" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:11:59.625128 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.625093    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-kube-api-access-jdfwm" (OuterVolumeSpecName: "kube-api-access-jdfwm") pod "58c018bb-8a6f-4e8f-8a94-51f9bf03a063" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063"). InnerVolumeSpecName "kube-api-access-jdfwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:11:59.625241 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.625171    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "58c018bb-8a6f-4e8f-8a94-51f9bf03a063" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:11:59.625314 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.625266    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "58c018bb-8a6f-4e8f-8a94-51f9bf03a063" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:11:59.625412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.625391    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "58c018bb-8a6f-4e8f-8a94-51f9bf03a063" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:11:59.625637 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.625616    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "58c018bb-8a6f-4e8f-8a94-51f9bf03a063" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:11:59.631535 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.631504    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "58c018bb-8a6f-4e8f-8a94-51f9bf03a063" (UID: "58c018bb-8a6f-4e8f-8a94-51f9bf03a063"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:11:59.723265 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.723230    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-installation-pull-secrets\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:11:59.723265 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.723261    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdfwm\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-kube-api-access-jdfwm\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:11:59.723265 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.723273    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-ca-trust-extracted\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:11:59.723466 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.723283    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-trusted-ca\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:11:59.723466 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.723292    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-bound-sa-token\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:11:59.723466 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.723300    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-certificates\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:11:59.723466 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.723312    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-image-registry-private-configuration\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:11:59.723466 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:11:59.723321    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c018bb-8a6f-4e8f-8a94-51f9bf03a063-registry-tls\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:12:00.007953 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:12:00.007855    2578 generic.go:356] "Generic (PLEG): container finished" podID="58c018bb-8a6f-4e8f-8a94-51f9bf03a063" containerID="85f0087485bb7feeaf7bcd647d7ea80dda11cc033e0bb8cdf3e9b239f342352b" exitCode=0
Jul 01 18:12:00.007953 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:12:00.007897    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl"
Jul 01 18:12:00.007953 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:12:00.007927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl" event={"ID":"58c018bb-8a6f-4e8f-8a94-51f9bf03a063","Type":"ContainerDied","Data":"85f0087485bb7feeaf7bcd647d7ea80dda11cc033e0bb8cdf3e9b239f342352b"}
Jul 01 18:12:00.008262 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:12:00.007976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bdccbb57-zl6tl" event={"ID":"58c018bb-8a6f-4e8f-8a94-51f9bf03a063","Type":"ContainerDied","Data":"b6385c5e2b98dd1701f86138c97405433edca63db88dc8861f14ceac461ab2dc"}
Jul 01 18:12:00.008262 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:12:00.008027    2578 scope.go:117] "RemoveContainer" containerID="85f0087485bb7feeaf7bcd647d7ea80dda11cc033e0bb8cdf3e9b239f342352b"
Jul 01 18:12:00.017149 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:12:00.017126    2578 scope.go:117] "RemoveContainer" containerID="85f0087485bb7feeaf7bcd647d7ea80dda11cc033e0bb8cdf3e9b239f342352b"
Jul 01 18:12:00.017437 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:12:00.017417    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85f0087485bb7feeaf7bcd647d7ea80dda11cc033e0bb8cdf3e9b239f342352b\": container with ID starting with 85f0087485bb7feeaf7bcd647d7ea80dda11cc033e0bb8cdf3e9b239f342352b not found: ID does not exist" containerID="85f0087485bb7feeaf7bcd647d7ea80dda11cc033e0bb8cdf3e9b239f342352b"
Jul 01 18:12:00.017487 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:12:00.017447    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85f0087485bb7feeaf7bcd647d7ea80dda11cc033e0bb8cdf3e9b239f342352b"} err="failed to get container status \"85f0087485bb7feeaf7bcd647d7ea80dda11cc033e0bb8cdf3e9b239f342352b\": rpc error: code = NotFound desc = could not find container \"85f0087485bb7feeaf7bcd647d7ea80dda11cc033e0bb8cdf3e9b239f342352b\": container with ID starting with 85f0087485bb7feeaf7bcd647d7ea80dda11cc033e0bb8cdf3e9b239f342352b not found: ID does not exist"
Jul 01 18:12:00.029370 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:12:00.029329    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55bdccbb57-zl6tl"]
Jul 01 18:12:00.035954 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:12:00.035922    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55bdccbb57-zl6tl"]
Jul 01 18:12:00.820511 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:12:00.820478    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58c018bb-8a6f-4e8f-8a94-51f9bf03a063" path="/var/lib/kubelet/pods/58c018bb-8a6f-4e8f-8a94-51f9bf03a063/volumes"
Jul 01 18:15:25.394459 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.394378    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-tx862"]
Jul 01 18:15:25.394900 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.394603    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c018bb-8a6f-4e8f-8a94-51f9bf03a063" containerName="registry"
Jul 01 18:15:25.394900 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.394613    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c018bb-8a6f-4e8f-8a94-51f9bf03a063" containerName="registry"
Jul 01 18:15:25.394900 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.394661    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c018bb-8a6f-4e8f-8a94-51f9bf03a063" containerName="registry"
Jul 01 18:15:25.397419 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.397403    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-tx862"
Jul 01 18:15:25.400533 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.400403    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 01 18:15:25.400533 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.400476    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 01 18:15:25.400722 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.400540    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 01 18:15:25.400722 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.400478    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 01 18:15:25.404867 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.404847    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-f2lv5\""
Jul 01 18:15:25.407158 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.407135    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-tx862"]
Jul 01 18:15:25.467865 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.467830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f98ca54c-1f65-436d-947a-a8893dc4da44-certificates\") pod \"keda-admission-5fbd5c4644-tx862\" (UID: \"f98ca54c-1f65-436d-947a-a8893dc4da44\") " pod="openshift-keda/keda-admission-5fbd5c4644-tx862"
Jul 01 18:15:25.467865 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.467877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnf5h\" (UniqueName: \"kubernetes.io/projected/f98ca54c-1f65-436d-947a-a8893dc4da44-kube-api-access-nnf5h\") pod \"keda-admission-5fbd5c4644-tx862\" (UID: \"f98ca54c-1f65-436d-947a-a8893dc4da44\") " pod="openshift-keda/keda-admission-5fbd5c4644-tx862"
Jul 01 18:15:25.568596 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.568559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f98ca54c-1f65-436d-947a-a8893dc4da44-certificates\") pod \"keda-admission-5fbd5c4644-tx862\" (UID: \"f98ca54c-1f65-436d-947a-a8893dc4da44\") " pod="openshift-keda/keda-admission-5fbd5c4644-tx862"
Jul 01 18:15:25.568596 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.568609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnf5h\" (UniqueName: \"kubernetes.io/projected/f98ca54c-1f65-436d-947a-a8893dc4da44-kube-api-access-nnf5h\") pod \"keda-admission-5fbd5c4644-tx862\" (UID: \"f98ca54c-1f65-436d-947a-a8893dc4da44\") " pod="openshift-keda/keda-admission-5fbd5c4644-tx862"
Jul 01 18:15:25.568818 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:15:25.568718    2578 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 01 18:15:25.568818 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:15:25.568747    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-tx862: secret "keda-admission-webhooks-certs" not found
Jul 01 18:15:25.568818 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:15:25.568813    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f98ca54c-1f65-436d-947a-a8893dc4da44-certificates podName:f98ca54c-1f65-436d-947a-a8893dc4da44 nodeName:}" failed. No retries permitted until 2026-07-01 18:15:26.068795147 +0000 UTC m=+547.636080119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f98ca54c-1f65-436d-947a-a8893dc4da44-certificates") pod "keda-admission-5fbd5c4644-tx862" (UID: "f98ca54c-1f65-436d-947a-a8893dc4da44") : secret "keda-admission-webhooks-certs" not found
Jul 01 18:15:25.577767 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:25.577729    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnf5h\" (UniqueName: \"kubernetes.io/projected/f98ca54c-1f65-436d-947a-a8893dc4da44-kube-api-access-nnf5h\") pod \"keda-admission-5fbd5c4644-tx862\" (UID: \"f98ca54c-1f65-436d-947a-a8893dc4da44\") " pod="openshift-keda/keda-admission-5fbd5c4644-tx862"
Jul 01 18:15:26.073937 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:26.073886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f98ca54c-1f65-436d-947a-a8893dc4da44-certificates\") pod \"keda-admission-5fbd5c4644-tx862\" (UID: \"f98ca54c-1f65-436d-947a-a8893dc4da44\") " pod="openshift-keda/keda-admission-5fbd5c4644-tx862"
Jul 01 18:15:26.076494 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:26.076464    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f98ca54c-1f65-436d-947a-a8893dc4da44-certificates\") pod \"keda-admission-5fbd5c4644-tx862\" (UID: \"f98ca54c-1f65-436d-947a-a8893dc4da44\") " pod="openshift-keda/keda-admission-5fbd5c4644-tx862"
Jul 01 18:15:26.313516 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:26.313473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-tx862"
Jul 01 18:15:26.442312 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:26.442279    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-tx862"]
Jul 01 18:15:26.447929 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:26.447907    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 18:15:26.689118 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:26.689020    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-tx862" event={"ID":"f98ca54c-1f65-436d-947a-a8893dc4da44","Type":"ContainerStarted","Data":"17fea736c716b17c77d1368eb7599e145ffdaa23781abbd04d6619eb2683dd8b"}
Jul 01 18:15:29.702251 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:29.702217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-tx862" event={"ID":"f98ca54c-1f65-436d-947a-a8893dc4da44","Type":"ContainerStarted","Data":"ebe69f33da01ce04bc893464b2bf3452f7bed7a12f766c67e7b4ab2d3988e8c3"}
Jul 01 18:15:29.702640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:29.702349    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-tx862"
Jul 01 18:15:29.719579 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:29.719529    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-tx862" podStartSLOduration=2.288020297 podStartE2EDuration="4.719511424s" totalImagesPullingTime="2.431491127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:15:29.718193879 +0000 UTC m=+551.285478858" watchObservedRunningTime="2026-07-01 18:15:29.719511424 +0000 UTC m=+551.286796401"
Jul 01 18:15:50.709792 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:15:50.709758    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-tx862"
Jul 01 18:16:18.783212 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:18.783167    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:16:18.783813 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:18.783352    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:16:29.933430 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:29.933394    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-vgv4w"]
Jul 01 18:16:29.936674 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:29.936655    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-vgv4w"
Jul 01 18:16:29.939163 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:29.939135    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 01 18:16:29.939345 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:29.939230    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 18:16:29.940308 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:29.940281    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-f8mrp\""
Jul 01 18:16:29.940415 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:29.940345    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 18:16:29.945470 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:29.945441    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-vgv4w"]
Jul 01 18:16:30.021434 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:30.021392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c96fe95-60ea-492e-91ac-9589e6316a7a-cert\") pod \"llmisvc-controller-manager-67d95fdc48-vgv4w\" (UID: \"0c96fe95-60ea-492e-91ac-9589e6316a7a\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-vgv4w"
Jul 01 18:16:30.021623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:30.021498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrsqm\" (UniqueName: \"kubernetes.io/projected/0c96fe95-60ea-492e-91ac-9589e6316a7a-kube-api-access-jrsqm\") pod \"llmisvc-controller-manager-67d95fdc48-vgv4w\" (UID: \"0c96fe95-60ea-492e-91ac-9589e6316a7a\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-vgv4w"
Jul 01 18:16:30.122751 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:30.122724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsqm\" (UniqueName: \"kubernetes.io/projected/0c96fe95-60ea-492e-91ac-9589e6316a7a-kube-api-access-jrsqm\") pod \"llmisvc-controller-manager-67d95fdc48-vgv4w\" (UID: \"0c96fe95-60ea-492e-91ac-9589e6316a7a\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-vgv4w"
Jul 01 18:16:30.122906 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:30.122773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c96fe95-60ea-492e-91ac-9589e6316a7a-cert\") pod \"llmisvc-controller-manager-67d95fdc48-vgv4w\" (UID: \"0c96fe95-60ea-492e-91ac-9589e6316a7a\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-vgv4w"
Jul 01 18:16:30.125259 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:30.125225    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c96fe95-60ea-492e-91ac-9589e6316a7a-cert\") pod \"llmisvc-controller-manager-67d95fdc48-vgv4w\" (UID: \"0c96fe95-60ea-492e-91ac-9589e6316a7a\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-vgv4w"
Jul 01 18:16:30.131366 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:30.131334    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrsqm\" (UniqueName: \"kubernetes.io/projected/0c96fe95-60ea-492e-91ac-9589e6316a7a-kube-api-access-jrsqm\") pod \"llmisvc-controller-manager-67d95fdc48-vgv4w\" (UID: \"0c96fe95-60ea-492e-91ac-9589e6316a7a\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-vgv4w"
Jul 01 18:16:30.247324 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:30.247223    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-vgv4w"
Jul 01 18:16:30.378970 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:30.378935    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-vgv4w"]
Jul 01 18:16:30.382602 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:16:30.382571    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0c96fe95_60ea_492e_91ac_9589e6316a7a.slice/crio-2a9a4925c321bf77d5fcf1089bcebb4913e5f02991b104f51d2d0aea66e6c9e5 WatchSource:0}: Error finding container 2a9a4925c321bf77d5fcf1089bcebb4913e5f02991b104f51d2d0aea66e6c9e5: Status 404 returned error can't find the container with id 2a9a4925c321bf77d5fcf1089bcebb4913e5f02991b104f51d2d0aea66e6c9e5
Jul 01 18:16:30.916769 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:30.916734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-vgv4w" event={"ID":"0c96fe95-60ea-492e-91ac-9589e6316a7a","Type":"ContainerStarted","Data":"2a9a4925c321bf77d5fcf1089bcebb4913e5f02991b104f51d2d0aea66e6c9e5"}
Jul 01 18:16:32.927357 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:32.927324    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-vgv4w" event={"ID":"0c96fe95-60ea-492e-91ac-9589e6316a7a","Type":"ContainerStarted","Data":"58640373556be602c3bf35479e49756238519065a2d68a25b6a6e2847bc5f3ff"}
Jul 01 18:16:32.927773 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:32.927477    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-vgv4w"
Jul 01 18:16:32.945782 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:16:32.945723    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-vgv4w" podStartSLOduration=2.127914126 podStartE2EDuration="3.945703621s" totalImagesPullingTime="1.817789495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:16:32.943666255 +0000 UTC m=+614.510951532" watchObservedRunningTime="2026-07-01 18:16:32.945703621 +0000 UTC m=+614.512988599"
Jul 01 18:17:03.935121 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:03.935029    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-vgv4w"
Jul 01 18:17:39.067542 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:39.067492    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zwmnb"]
Jul 01 18:17:39.071173 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:39.071150    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-zwmnb"
Jul 01 18:17:39.074024 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:39.073983    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-w8z6v\""
Jul 01 18:17:39.075476 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:39.075459    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 01 18:17:39.080506 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:39.080480    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zwmnb"]
Jul 01 18:17:39.130083 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:39.130044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7359dbfd-6c9f-405a-aa42-603da2bbc997-cert\") pod \"odh-model-controller-84b54b7f97-zwmnb\" (UID: \"7359dbfd-6c9f-405a-aa42-603da2bbc997\") " pod="kserve/odh-model-controller-84b54b7f97-zwmnb"
Jul 01 18:17:39.130250 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:39.130092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpvkb\" (UniqueName: \"kubernetes.io/projected/7359dbfd-6c9f-405a-aa42-603da2bbc997-kube-api-access-vpvkb\") pod \"odh-model-controller-84b54b7f97-zwmnb\" (UID: \"7359dbfd-6c9f-405a-aa42-603da2bbc997\") " pod="kserve/odh-model-controller-84b54b7f97-zwmnb"
Jul 01 18:17:39.231022 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:39.230952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7359dbfd-6c9f-405a-aa42-603da2bbc997-cert\") pod \"odh-model-controller-84b54b7f97-zwmnb\" (UID: \"7359dbfd-6c9f-405a-aa42-603da2bbc997\") " pod="kserve/odh-model-controller-84b54b7f97-zwmnb"
Jul 01 18:17:39.231022 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:39.231032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpvkb\" (UniqueName: \"kubernetes.io/projected/7359dbfd-6c9f-405a-aa42-603da2bbc997-kube-api-access-vpvkb\") pod \"odh-model-controller-84b54b7f97-zwmnb\" (UID: \"7359dbfd-6c9f-405a-aa42-603da2bbc997\") " pod="kserve/odh-model-controller-84b54b7f97-zwmnb"
Jul 01 18:17:39.233513 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:39.233481    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7359dbfd-6c9f-405a-aa42-603da2bbc997-cert\") pod \"odh-model-controller-84b54b7f97-zwmnb\" (UID: \"7359dbfd-6c9f-405a-aa42-603da2bbc997\") " pod="kserve/odh-model-controller-84b54b7f97-zwmnb"
Jul 01 18:17:39.239948 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:39.239918    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpvkb\" (UniqueName: \"kubernetes.io/projected/7359dbfd-6c9f-405a-aa42-603da2bbc997-kube-api-access-vpvkb\") pod \"odh-model-controller-84b54b7f97-zwmnb\" (UID: \"7359dbfd-6c9f-405a-aa42-603da2bbc997\") " pod="kserve/odh-model-controller-84b54b7f97-zwmnb"
Jul 01 18:17:39.382274 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:39.382171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-zwmnb"
Jul 01 18:17:39.513389 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:39.513355    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zwmnb"]
Jul 01 18:17:39.516217 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:17:39.516188    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7359dbfd_6c9f_405a_aa42_603da2bbc997.slice/crio-4a01f613d76ea901b54864780a0eb2089ca18ca52406a7e72e8ed8c07bc15adb WatchSource:0}: Error finding container 4a01f613d76ea901b54864780a0eb2089ca18ca52406a7e72e8ed8c07bc15adb: Status 404 returned error can't find the container with id 4a01f613d76ea901b54864780a0eb2089ca18ca52406a7e72e8ed8c07bc15adb
Jul 01 18:17:40.158435 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:40.158394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-zwmnb" event={"ID":"7359dbfd-6c9f-405a-aa42-603da2bbc997","Type":"ContainerStarted","Data":"4a01f613d76ea901b54864780a0eb2089ca18ca52406a7e72e8ed8c07bc15adb"}
Jul 01 18:17:44.175328 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:44.175239    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-zwmnb" event={"ID":"7359dbfd-6c9f-405a-aa42-603da2bbc997","Type":"ContainerStarted","Data":"afccf459c1b28d9e45b0f5450f6b089a80dc40ae7372a6a3e70a20afce0002c3"}
Jul 01 18:17:44.175779 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:44.175363    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-zwmnb"
Jul 01 18:17:44.191373 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:44.191318    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-zwmnb" podStartSLOduration=1.194966881 podStartE2EDuration="5.19130246s" totalImagesPullingTime="3.996335579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:17:44.191032913 +0000 UTC m=+685.758317889" watchObservedRunningTime="2026-07-01 18:17:44.19130246 +0000 UTC m=+685.758587480"
Jul 01 18:17:55.183100 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:17:55.183070    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-zwmnb"
Jul 01 18:19:27.949632 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:27.949597    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"]
Jul 01 18:19:27.952838 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:27.952821    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:19:27.955061 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:27.955037    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4d812-kube-rbac-proxy-sar-config\""
Jul 01 18:19:27.955201 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:27.955041    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4d812-predictor-serving-cert\""
Jul 01 18:19:27.955201 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:27.955194    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 18:19:27.956191 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:27.956175    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fg8rn\""
Jul 01 18:19:27.956292 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:27.956177    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 18:19:27.962079 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:27.962055    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"]
Jul 01 18:19:28.026482 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:28.026448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b988bd0-5e2e-44de-80d1-3bde2076b377-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4d812-predictor-cf946d449-cpmt9\" (UID: \"8b988bd0-5e2e-44de-80d1-3bde2076b377\") " pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:19:28.026652 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:28.026495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5f4t\" (UniqueName: \"kubernetes.io/projected/8b988bd0-5e2e-44de-80d1-3bde2076b377-kube-api-access-k5f4t\") pod \"error-404-isvc-4d812-predictor-cf946d449-cpmt9\" (UID: \"8b988bd0-5e2e-44de-80d1-3bde2076b377\") " pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:19:28.026652 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:28.026562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b988bd0-5e2e-44de-80d1-3bde2076b377-proxy-tls\") pod \"error-404-isvc-4d812-predictor-cf946d449-cpmt9\" (UID: \"8b988bd0-5e2e-44de-80d1-3bde2076b377\") " pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:19:28.127379 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:28.127342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b988bd0-5e2e-44de-80d1-3bde2076b377-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4d812-predictor-cf946d449-cpmt9\" (UID: \"8b988bd0-5e2e-44de-80d1-3bde2076b377\") " pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:19:28.127557 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:28.127400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5f4t\" (UniqueName: \"kubernetes.io/projected/8b988bd0-5e2e-44de-80d1-3bde2076b377-kube-api-access-k5f4t\") pod \"error-404-isvc-4d812-predictor-cf946d449-cpmt9\" (UID: \"8b988bd0-5e2e-44de-80d1-3bde2076b377\") " pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:19:28.127557 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:28.127436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b988bd0-5e2e-44de-80d1-3bde2076b377-proxy-tls\") pod \"error-404-isvc-4d812-predictor-cf946d449-cpmt9\" (UID: \"8b988bd0-5e2e-44de-80d1-3bde2076b377\") " pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:19:28.128151 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:28.128123    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b988bd0-5e2e-44de-80d1-3bde2076b377-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4d812-predictor-cf946d449-cpmt9\" (UID: \"8b988bd0-5e2e-44de-80d1-3bde2076b377\") " pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:19:28.130065 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:28.130040    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b988bd0-5e2e-44de-80d1-3bde2076b377-proxy-tls\") pod \"error-404-isvc-4d812-predictor-cf946d449-cpmt9\" (UID: \"8b988bd0-5e2e-44de-80d1-3bde2076b377\") " pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:19:28.141684 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:28.141653    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5f4t\" (UniqueName: \"kubernetes.io/projected/8b988bd0-5e2e-44de-80d1-3bde2076b377-kube-api-access-k5f4t\") pod \"error-404-isvc-4d812-predictor-cf946d449-cpmt9\" (UID: \"8b988bd0-5e2e-44de-80d1-3bde2076b377\") " pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:19:28.263789 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:28.263703    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:19:28.395849 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:28.395815    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"]
Jul 01 18:19:28.398907 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:19:28.398873    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b988bd0_5e2e_44de_80d1_3bde2076b377.slice/crio-c8de650b523762c4cc6d0841eb10de58e049d5285eb4f0c831edd917d4b0d20a WatchSource:0}: Error finding container c8de650b523762c4cc6d0841eb10de58e049d5285eb4f0c831edd917d4b0d20a: Status 404 returned error can't find the container with id c8de650b523762c4cc6d0841eb10de58e049d5285eb4f0c831edd917d4b0d20a
Jul 01 18:19:28.549401 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:28.549370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" event={"ID":"8b988bd0-5e2e-44de-80d1-3bde2076b377","Type":"ContainerStarted","Data":"c8de650b523762c4cc6d0841eb10de58e049d5285eb4f0c831edd917d4b0d20a"}
Jul 01 18:19:41.606598 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:41.606564    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" event={"ID":"8b988bd0-5e2e-44de-80d1-3bde2076b377","Type":"ContainerStarted","Data":"f06955abf27db4352cd14700267e0ce00a6b4e809cf9e4850b0a3e6a6023d2b8"}
Jul 01 18:19:44.621941 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:44.621843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" event={"ID":"8b988bd0-5e2e-44de-80d1-3bde2076b377","Type":"ContainerStarted","Data":"e80c90220eb107d8027c643ae8d7b7845362dd1616a1c1af0709829ff860e99a"}
Jul 01 18:19:44.621941 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:44.621918    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:19:44.622446 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:44.621950    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:19:44.623421 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:44.623393    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Jul 01 18:19:44.639402 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:44.639341    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" podStartSLOduration=1.817144452 podStartE2EDuration="17.639323664s" totalImagesPullingTime="15.822179212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:19:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:19:44.637888966 +0000 UTC m=+806.205173945" watchObservedRunningTime="2026-07-01 18:19:44.639323664 +0000 UTC m=+806.206608642"
Jul 01 18:19:45.625275 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:45.625236    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Jul 01 18:19:50.631163 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:50.631134    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:19:50.631656 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:19:50.631627    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Jul 01 18:20:00.631871 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:20:00.631829    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Jul 01 18:20:10.632652 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:20:10.632594    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Jul 01 18:20:20.631704 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:20:20.631659    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Jul 01 18:20:30.633186 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:20:30.633151    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:21:02.169462 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.169426    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"]
Jul 01 18:21:02.172730 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.172710    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:02.175252 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.175229    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2588a-kube-rbac-proxy-sar-config\""
Jul 01 18:21:02.175511 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.175481    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2588a-predictor-serving-cert\""
Jul 01 18:21:02.183516 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.183488    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"]
Jul 01 18:21:02.224529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.224487    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"]
Jul 01 18:21:02.224811 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.224784    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kserve-container" containerID="cri-o://f06955abf27db4352cd14700267e0ce00a6b4e809cf9e4850b0a3e6a6023d2b8" gracePeriod=30
Jul 01 18:21:02.224885 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.224806    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kube-rbac-proxy" containerID="cri-o://e80c90220eb107d8027c643ae8d7b7845362dd1616a1c1af0709829ff860e99a" gracePeriod=30
Jul 01 18:21:02.251158 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.251114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7p2m\" (UniqueName: \"kubernetes.io/projected/6d05b78c-7dca-4f07-b37a-007729f90494-kube-api-access-d7p2m\") pod \"error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8\" (UID: \"6d05b78c-7dca-4f07-b37a-007729f90494\") " pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:02.251158 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.251162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d05b78c-7dca-4f07-b37a-007729f90494-proxy-tls\") pod \"error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8\" (UID: \"6d05b78c-7dca-4f07-b37a-007729f90494\") " pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:02.251357 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.251187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d05b78c-7dca-4f07-b37a-007729f90494-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8\" (UID: \"6d05b78c-7dca-4f07-b37a-007729f90494\") " pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:02.351698 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.351657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7p2m\" (UniqueName: \"kubernetes.io/projected/6d05b78c-7dca-4f07-b37a-007729f90494-kube-api-access-d7p2m\") pod \"error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8\" (UID: \"6d05b78c-7dca-4f07-b37a-007729f90494\") " pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:02.351808 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.351711    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d05b78c-7dca-4f07-b37a-007729f90494-proxy-tls\") pod \"error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8\" (UID: \"6d05b78c-7dca-4f07-b37a-007729f90494\") " pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:02.351808 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.351740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d05b78c-7dca-4f07-b37a-007729f90494-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8\" (UID: \"6d05b78c-7dca-4f07-b37a-007729f90494\") " pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:02.351910 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:21:02.351859    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-2588a-predictor-serving-cert: secret "error-404-isvc-2588a-predictor-serving-cert" not found
Jul 01 18:21:02.351960 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:21:02.351948    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d05b78c-7dca-4f07-b37a-007729f90494-proxy-tls podName:6d05b78c-7dca-4f07-b37a-007729f90494 nodeName:}" failed. No retries permitted until 2026-07-01 18:21:02.851927388 +0000 UTC m=+884.419212351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6d05b78c-7dca-4f07-b37a-007729f90494-proxy-tls") pod "error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" (UID: "6d05b78c-7dca-4f07-b37a-007729f90494") : secret "error-404-isvc-2588a-predictor-serving-cert" not found
Jul 01 18:21:02.352446 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.352429    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d05b78c-7dca-4f07-b37a-007729f90494-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8\" (UID: \"6d05b78c-7dca-4f07-b37a-007729f90494\") " pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:02.361078 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.361043    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7p2m\" (UniqueName: \"kubernetes.io/projected/6d05b78c-7dca-4f07-b37a-007729f90494-kube-api-access-d7p2m\") pod \"error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8\" (UID: \"6d05b78c-7dca-4f07-b37a-007729f90494\") " pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:02.857250 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.857212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d05b78c-7dca-4f07-b37a-007729f90494-proxy-tls\") pod \"error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8\" (UID: \"6d05b78c-7dca-4f07-b37a-007729f90494\") " pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:02.859826 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.859799    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d05b78c-7dca-4f07-b37a-007729f90494-proxy-tls\") pod \"error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8\" (UID: \"6d05b78c-7dca-4f07-b37a-007729f90494\") " pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:02.901454 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.901417    2578 generic.go:356] "Generic (PLEG): container finished" podID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerID="e80c90220eb107d8027c643ae8d7b7845362dd1616a1c1af0709829ff860e99a" exitCode=2
Jul 01 18:21:02.901619 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:02.901486    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" event={"ID":"8b988bd0-5e2e-44de-80d1-3bde2076b377","Type":"ContainerDied","Data":"e80c90220eb107d8027c643ae8d7b7845362dd1616a1c1af0709829ff860e99a"}
Jul 01 18:21:03.083426 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:03.083376    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:03.217649 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:03.217611    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"]
Jul 01 18:21:03.220745 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:21:03.220720    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d05b78c_7dca_4f07_b37a_007729f90494.slice/crio-4a439dea794bec1de42171ac32666407fb0aa38bd984513128a907cb2e728783 WatchSource:0}: Error finding container 4a439dea794bec1de42171ac32666407fb0aa38bd984513128a907cb2e728783: Status 404 returned error can't find the container with id 4a439dea794bec1de42171ac32666407fb0aa38bd984513128a907cb2e728783
Jul 01 18:21:03.906713 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:03.906677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" event={"ID":"6d05b78c-7dca-4f07-b37a-007729f90494","Type":"ContainerStarted","Data":"f931d1246c444b4e1f6c277a3122063cf5be9e150587152077d73c34d179a88e"}
Jul 01 18:21:03.906866 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:03.906734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" event={"ID":"6d05b78c-7dca-4f07-b37a-007729f90494","Type":"ContainerStarted","Data":"41cb51740db21fa8a7677ad38358ef87f3e66684bd801549a1e517adc4f51f93"}
Jul 01 18:21:03.906866 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:03.906755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" event={"ID":"6d05b78c-7dca-4f07-b37a-007729f90494","Type":"ContainerStarted","Data":"4a439dea794bec1de42171ac32666407fb0aa38bd984513128a907cb2e728783"}
Jul 01 18:21:03.906866 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:03.906814    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:03.925129 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:03.925076    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" podStartSLOduration=1.925058921 podStartE2EDuration="1.925058921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:21:03.92337404 +0000 UTC m=+885.490659018" watchObservedRunningTime="2026-07-01 18:21:03.925058921 +0000 UTC m=+885.492343900"
Jul 01 18:21:04.910178 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:04.910140    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:04.911497 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:04.911467    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 01 18:21:05.625933 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:05.625884    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
Jul 01 18:21:05.915228 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:05.915194    2578 generic.go:356] "Generic (PLEG): container finished" podID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerID="f06955abf27db4352cd14700267e0ce00a6b4e809cf9e4850b0a3e6a6023d2b8" exitCode=0
Jul 01 18:21:05.915614 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:05.915265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" event={"ID":"8b988bd0-5e2e-44de-80d1-3bde2076b377","Type":"ContainerDied","Data":"f06955abf27db4352cd14700267e0ce00a6b4e809cf9e4850b0a3e6a6023d2b8"}
Jul 01 18:21:05.915663 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:05.915601    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 01 18:21:06.069183 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.069155    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:21:06.181723 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.181620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b988bd0-5e2e-44de-80d1-3bde2076b377-kube-rbac-proxy-sar-config\") pod \"8b988bd0-5e2e-44de-80d1-3bde2076b377\" (UID: \"8b988bd0-5e2e-44de-80d1-3bde2076b377\") "
Jul 01 18:21:06.181723 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.181689    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b988bd0-5e2e-44de-80d1-3bde2076b377-proxy-tls\") pod \"8b988bd0-5e2e-44de-80d1-3bde2076b377\" (UID: \"8b988bd0-5e2e-44de-80d1-3bde2076b377\") "
Jul 01 18:21:06.181723 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.181722    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5f4t\" (UniqueName: \"kubernetes.io/projected/8b988bd0-5e2e-44de-80d1-3bde2076b377-kube-api-access-k5f4t\") pod \"8b988bd0-5e2e-44de-80d1-3bde2076b377\" (UID: \"8b988bd0-5e2e-44de-80d1-3bde2076b377\") "
Jul 01 18:21:06.182117 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.182092    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b988bd0-5e2e-44de-80d1-3bde2076b377-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b988bd0-5e2e-44de-80d1-3bde2076b377" (UID: "8b988bd0-5e2e-44de-80d1-3bde2076b377"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:21:06.184122 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.184088    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b988bd0-5e2e-44de-80d1-3bde2076b377-kube-api-access-k5f4t" (OuterVolumeSpecName: "kube-api-access-k5f4t") pod "8b988bd0-5e2e-44de-80d1-3bde2076b377" (UID: "8b988bd0-5e2e-44de-80d1-3bde2076b377"). InnerVolumeSpecName "kube-api-access-k5f4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:21:06.184122 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.184090    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b988bd0-5e2e-44de-80d1-3bde2076b377-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b988bd0-5e2e-44de-80d1-3bde2076b377" (UID: "8b988bd0-5e2e-44de-80d1-3bde2076b377"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:21:06.282904 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.282851    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b988bd0-5e2e-44de-80d1-3bde2076b377-proxy-tls\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:21:06.282904 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.282896    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5f4t\" (UniqueName: \"kubernetes.io/projected/8b988bd0-5e2e-44de-80d1-3bde2076b377-kube-api-access-k5f4t\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:21:06.282904 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.282906    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b988bd0-5e2e-44de-80d1-3bde2076b377-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:21:06.920297 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.920267    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"
Jul 01 18:21:06.920297 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.920279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9" event={"ID":"8b988bd0-5e2e-44de-80d1-3bde2076b377","Type":"ContainerDied","Data":"c8de650b523762c4cc6d0841eb10de58e049d5285eb4f0c831edd917d4b0d20a"}
Jul 01 18:21:06.920769 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.920344    2578 scope.go:117] "RemoveContainer" containerID="e80c90220eb107d8027c643ae8d7b7845362dd1616a1c1af0709829ff860e99a"
Jul 01 18:21:06.929104 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.929079    2578 scope.go:117] "RemoveContainer" containerID="f06955abf27db4352cd14700267e0ce00a6b4e809cf9e4850b0a3e6a6023d2b8"
Jul 01 18:21:06.938323 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.938298    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"]
Jul 01 18:21:06.943205 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:06.943181    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4d812-predictor-cf946d449-cpmt9"]
Jul 01 18:21:08.820298 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:08.820263    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" path="/var/lib/kubelet/pods/8b988bd0-5e2e-44de-80d1-3bde2076b377/volumes"
Jul 01 18:21:10.920110 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:10.920083    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:10.920682 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:10.920652    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 01 18:21:18.803152 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:18.803122    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:21:18.804089 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:18.804065    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:21:20.920565 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:20.920517    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 01 18:21:30.921113 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:30.921071    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 01 18:21:40.921124 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:40.921032    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 01 18:21:50.922186 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:50.922152    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:21:58.009261 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.009221    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"]
Jul 01 18:21:58.009982 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.009492    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kserve-container"
Jul 01 18:21:58.009982 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.009503    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kserve-container"
Jul 01 18:21:58.009982 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.009517    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kube-rbac-proxy"
Jul 01 18:21:58.009982 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.009523    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kube-rbac-proxy"
Jul 01 18:21:58.009982 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.009588    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kube-rbac-proxy"
Jul 01 18:21:58.009982 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.009598    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b988bd0-5e2e-44de-80d1-3bde2076b377" containerName="kserve-container"
Jul 01 18:21:58.012479 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.012461    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:21:58.014862 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.014836    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c5c10-kube-rbac-proxy-sar-config\""
Jul 01 18:21:58.015027 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.014881    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c5c10-predictor-serving-cert\""
Jul 01 18:21:58.020479 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.020457    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"]
Jul 01 18:21:58.082862 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.082822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnggd\" (UniqueName: \"kubernetes.io/projected/6727f26d-7a38-47ac-95b0-74d409638145-kube-api-access-jnggd\") pod \"error-404-isvc-c5c10-predictor-dd55b8f87-dncv2\" (UID: \"6727f26d-7a38-47ac-95b0-74d409638145\") " pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:21:58.083088 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.082887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6727f26d-7a38-47ac-95b0-74d409638145-proxy-tls\") pod \"error-404-isvc-c5c10-predictor-dd55b8f87-dncv2\" (UID: \"6727f26d-7a38-47ac-95b0-74d409638145\") " pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:21:58.083088 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.082931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6727f26d-7a38-47ac-95b0-74d409638145-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c5c10-predictor-dd55b8f87-dncv2\" (UID: \"6727f26d-7a38-47ac-95b0-74d409638145\") " pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:21:58.184027 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.183980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnggd\" (UniqueName: \"kubernetes.io/projected/6727f26d-7a38-47ac-95b0-74d409638145-kube-api-access-jnggd\") pod \"error-404-isvc-c5c10-predictor-dd55b8f87-dncv2\" (UID: \"6727f26d-7a38-47ac-95b0-74d409638145\") " pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:21:58.184227 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.184043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6727f26d-7a38-47ac-95b0-74d409638145-proxy-tls\") pod \"error-404-isvc-c5c10-predictor-dd55b8f87-dncv2\" (UID: \"6727f26d-7a38-47ac-95b0-74d409638145\") " pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:21:58.184227 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.184074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6727f26d-7a38-47ac-95b0-74d409638145-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c5c10-predictor-dd55b8f87-dncv2\" (UID: \"6727f26d-7a38-47ac-95b0-74d409638145\") " pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:21:58.184227 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:21:58.184203    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-serving-cert: secret "error-404-isvc-c5c10-predictor-serving-cert" not found
Jul 01 18:21:58.184390 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:21:58.184279    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6727f26d-7a38-47ac-95b0-74d409638145-proxy-tls podName:6727f26d-7a38-47ac-95b0-74d409638145 nodeName:}" failed. No retries permitted until 2026-07-01 18:21:58.684256531 +0000 UTC m=+940.251541488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6727f26d-7a38-47ac-95b0-74d409638145-proxy-tls") pod "error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" (UID: "6727f26d-7a38-47ac-95b0-74d409638145") : secret "error-404-isvc-c5c10-predictor-serving-cert" not found
Jul 01 18:21:58.184723 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.184703    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6727f26d-7a38-47ac-95b0-74d409638145-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c5c10-predictor-dd55b8f87-dncv2\" (UID: \"6727f26d-7a38-47ac-95b0-74d409638145\") " pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:21:58.192369 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.192340    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnggd\" (UniqueName: \"kubernetes.io/projected/6727f26d-7a38-47ac-95b0-74d409638145-kube-api-access-jnggd\") pod \"error-404-isvc-c5c10-predictor-dd55b8f87-dncv2\" (UID: \"6727f26d-7a38-47ac-95b0-74d409638145\") " pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:21:58.687197 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.687158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6727f26d-7a38-47ac-95b0-74d409638145-proxy-tls\") pod \"error-404-isvc-c5c10-predictor-dd55b8f87-dncv2\" (UID: \"6727f26d-7a38-47ac-95b0-74d409638145\") " pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:21:58.689631 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.689608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6727f26d-7a38-47ac-95b0-74d409638145-proxy-tls\") pod \"error-404-isvc-c5c10-predictor-dd55b8f87-dncv2\" (UID: \"6727f26d-7a38-47ac-95b0-74d409638145\") " pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:21:58.924244 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:58.924203    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:21:59.068161 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:59.068124    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"]
Jul 01 18:21:59.070970 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:21:59.070935    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6727f26d_7a38_47ac_95b0_74d409638145.slice/crio-5344a18dc21b32136158e0d600dde1b3c7d9a3762073d13b1ed8466a40ba02ea WatchSource:0}: Error finding container 5344a18dc21b32136158e0d600dde1b3c7d9a3762073d13b1ed8466a40ba02ea: Status 404 returned error can't find the container with id 5344a18dc21b32136158e0d600dde1b3c7d9a3762073d13b1ed8466a40ba02ea
Jul 01 18:21:59.115899 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:21:59.115869    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" event={"ID":"6727f26d-7a38-47ac-95b0-74d409638145","Type":"ContainerStarted","Data":"5344a18dc21b32136158e0d600dde1b3c7d9a3762073d13b1ed8466a40ba02ea"}
Jul 01 18:22:00.121587 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:22:00.121548    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" event={"ID":"6727f26d-7a38-47ac-95b0-74d409638145","Type":"ContainerStarted","Data":"f693b049141f4e67eead2066feeb4a03f00969c92b0d99a61990b97dbc19299d"}
Jul 01 18:22:00.122069 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:22:00.121744    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:22:00.122069 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:22:00.121795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" event={"ID":"6727f26d-7a38-47ac-95b0-74d409638145","Type":"ContainerStarted","Data":"2182d49e661599e7aca96c4a22d62225fedf9d74b762e4ad4d7a0c72448d7fa8"}
Jul 01 18:22:00.122069 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:22:00.121838    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:22:00.122925 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:22:00.122901    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 01 18:22:00.139053 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:22:00.138982    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" podStartSLOduration=3.138965759 podStartE2EDuration="3.138965759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:22:00.136983925 +0000 UTC m=+941.704268909" watchObservedRunningTime="2026-07-01 18:22:00.138965759 +0000 UTC m=+941.706250738"
Jul 01 18:22:01.125460 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:22:01.125406    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 01 18:22:06.131511 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:22:06.131479    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:22:06.132157 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:22:06.132123    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 01 18:22:16.132271 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:22:16.132229    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 01 18:22:26.132288 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:22:26.132241    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 01 18:22:36.132927 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:22:36.132876    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 01 18:22:46.133229 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:22:46.133137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:26:18.825333 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:26:18.825291    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:26:18.826438 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:26:18.826414    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:30:16.981500 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:16.981409    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"]
Jul 01 18:30:16.984797 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:16.984774    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:30:16.987972 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:16.987944    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0d3ed-predictor-serving-cert\""
Jul 01 18:30:16.987972 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:16.987944    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0d3ed-kube-rbac-proxy-sar-config\""
Jul 01 18:30:16.995981 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:16.995934    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"]
Jul 01 18:30:17.044582 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.044539    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"]
Jul 01 18:30:17.044841 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.044813    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" containerName="kserve-container" containerID="cri-o://41cb51740db21fa8a7677ad38358ef87f3e66684bd801549a1e517adc4f51f93" gracePeriod=30
Jul 01 18:30:17.044949 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.044883    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" containerName="kube-rbac-proxy" containerID="cri-o://f931d1246c444b4e1f6c277a3122063cf5be9e150587152077d73c34d179a88e" gracePeriod=30
Jul 01 18:30:17.085901 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.085853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48dac5a4-feb9-4052-9d7d-120cb2555d0e-proxy-tls\") pod \"error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd\" (UID: \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\") " pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:30:17.086117 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.085919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps2xl\" (UniqueName: \"kubernetes.io/projected/48dac5a4-feb9-4052-9d7d-120cb2555d0e-kube-api-access-ps2xl\") pod \"error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd\" (UID: \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\") " pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:30:17.086117 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.085951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48dac5a4-feb9-4052-9d7d-120cb2555d0e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd\" (UID: \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\") " pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:30:17.187207 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.187167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48dac5a4-feb9-4052-9d7d-120cb2555d0e-proxy-tls\") pod \"error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd\" (UID: \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\") " pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:30:17.187412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.187222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps2xl\" (UniqueName: \"kubernetes.io/projected/48dac5a4-feb9-4052-9d7d-120cb2555d0e-kube-api-access-ps2xl\") pod \"error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd\" (UID: \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\") " pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:30:17.187412 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.187245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48dac5a4-feb9-4052-9d7d-120cb2555d0e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd\" (UID: \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\") " pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:30:17.187943 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.187924    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48dac5a4-feb9-4052-9d7d-120cb2555d0e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd\" (UID: \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\") " pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:30:17.189642 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.189617    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48dac5a4-feb9-4052-9d7d-120cb2555d0e-proxy-tls\") pod \"error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd\" (UID: \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\") " pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:30:17.196015 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.195964    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps2xl\" (UniqueName: \"kubernetes.io/projected/48dac5a4-feb9-4052-9d7d-120cb2555d0e-kube-api-access-ps2xl\") pod \"error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd\" (UID: \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\") " pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:30:17.297228 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.297188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:30:17.438975 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.438768    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"]
Jul 01 18:30:17.442061 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:30:17.441978    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48dac5a4_feb9_4052_9d7d_120cb2555d0e.slice/crio-8e8b446a6e88b47fe591730df3b7a62a078b7c15c043e7c3869215ff16c25d52 WatchSource:0}: Error finding container 8e8b446a6e88b47fe591730df3b7a62a078b7c15c043e7c3869215ff16c25d52: Status 404 returned error can't find the container with id 8e8b446a6e88b47fe591730df3b7a62a078b7c15c043e7c3869215ff16c25d52
Jul 01 18:30:17.933183 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.933067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" event={"ID":"48dac5a4-feb9-4052-9d7d-120cb2555d0e","Type":"ContainerStarted","Data":"15d1fc73e81c090beb0cf2b4ff3141ce7d6e3033581d031d1c2e95a57d035cc9"}
Jul 01 18:30:17.933183 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.933149    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" event={"ID":"48dac5a4-feb9-4052-9d7d-120cb2555d0e","Type":"ContainerStarted","Data":"905f32789ede38d086b051191c7d231567b1c9549cfbd7c0c93960c75d284e01"}
Jul 01 18:30:17.933183 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.933179    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" event={"ID":"48dac5a4-feb9-4052-9d7d-120cb2555d0e","Type":"ContainerStarted","Data":"8e8b446a6e88b47fe591730df3b7a62a078b7c15c043e7c3869215ff16c25d52"}
Jul 01 18:30:17.933484 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.933210    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:30:17.935007 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.934962    2578 generic.go:356] "Generic (PLEG): container finished" podID="6d05b78c-7dca-4f07-b37a-007729f90494" containerID="f931d1246c444b4e1f6c277a3122063cf5be9e150587152077d73c34d179a88e" exitCode=2
Jul 01 18:30:17.935149 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.935030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" event={"ID":"6d05b78c-7dca-4f07-b37a-007729f90494","Type":"ContainerDied","Data":"f931d1246c444b4e1f6c277a3122063cf5be9e150587152077d73c34d179a88e"}
Jul 01 18:30:17.952426 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:17.952372    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" podStartSLOduration=1.952353332 podStartE2EDuration="1.952353332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:30:17.951938407 +0000 UTC m=+1439.519223400" watchObservedRunningTime="2026-07-01 18:30:17.952353332 +0000 UTC m=+1439.519638311"
Jul 01 18:30:18.939207 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:18.939172    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:30:18.940481 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:18.940449    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 18:30:19.942952 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:19.942910    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 18:30:20.695814 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.695784    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:30:20.818460 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.818377    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7p2m\" (UniqueName: \"kubernetes.io/projected/6d05b78c-7dca-4f07-b37a-007729f90494-kube-api-access-d7p2m\") pod \"6d05b78c-7dca-4f07-b37a-007729f90494\" (UID: \"6d05b78c-7dca-4f07-b37a-007729f90494\") "
Jul 01 18:30:20.818460 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.818432    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d05b78c-7dca-4f07-b37a-007729f90494-proxy-tls\") pod \"6d05b78c-7dca-4f07-b37a-007729f90494\" (UID: \"6d05b78c-7dca-4f07-b37a-007729f90494\") "
Jul 01 18:30:20.818646 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.818460    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d05b78c-7dca-4f07-b37a-007729f90494-kube-rbac-proxy-sar-config\") pod \"6d05b78c-7dca-4f07-b37a-007729f90494\" (UID: \"6d05b78c-7dca-4f07-b37a-007729f90494\") "
Jul 01 18:30:20.818975 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.818941    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d05b78c-7dca-4f07-b37a-007729f90494-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d05b78c-7dca-4f07-b37a-007729f90494" (UID: "6d05b78c-7dca-4f07-b37a-007729f90494"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:30:20.820649 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.820623    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d05b78c-7dca-4f07-b37a-007729f90494-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d05b78c-7dca-4f07-b37a-007729f90494" (UID: "6d05b78c-7dca-4f07-b37a-007729f90494"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:30:20.820957 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.820926    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d05b78c-7dca-4f07-b37a-007729f90494-kube-api-access-d7p2m" (OuterVolumeSpecName: "kube-api-access-d7p2m") pod "6d05b78c-7dca-4f07-b37a-007729f90494" (UID: "6d05b78c-7dca-4f07-b37a-007729f90494"). InnerVolumeSpecName "kube-api-access-d7p2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:30:20.919210 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.919168    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d05b78c-7dca-4f07-b37a-007729f90494-proxy-tls\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:30:20.919210 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.919203    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d05b78c-7dca-4f07-b37a-007729f90494-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:30:20.919210 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.919212    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7p2m\" (UniqueName: \"kubernetes.io/projected/6d05b78c-7dca-4f07-b37a-007729f90494-kube-api-access-d7p2m\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:30:20.947746 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.947709    2578 generic.go:356] "Generic (PLEG): container finished" podID="6d05b78c-7dca-4f07-b37a-007729f90494" containerID="41cb51740db21fa8a7677ad38358ef87f3e66684bd801549a1e517adc4f51f93" exitCode=0
Jul 01 18:30:20.948240 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.947763    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"
Jul 01 18:30:20.948240 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.947767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" event={"ID":"6d05b78c-7dca-4f07-b37a-007729f90494","Type":"ContainerDied","Data":"41cb51740db21fa8a7677ad38358ef87f3e66684bd801549a1e517adc4f51f93"}
Jul 01 18:30:20.948240 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.947851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8" event={"ID":"6d05b78c-7dca-4f07-b37a-007729f90494","Type":"ContainerDied","Data":"4a439dea794bec1de42171ac32666407fb0aa38bd984513128a907cb2e728783"}
Jul 01 18:30:20.948240 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.947895    2578 scope.go:117] "RemoveContainer" containerID="f931d1246c444b4e1f6c277a3122063cf5be9e150587152077d73c34d179a88e"
Jul 01 18:30:20.956238 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.956215    2578 scope.go:117] "RemoveContainer" containerID="41cb51740db21fa8a7677ad38358ef87f3e66684bd801549a1e517adc4f51f93"
Jul 01 18:30:20.965833 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.965813    2578 scope.go:117] "RemoveContainer" containerID="f931d1246c444b4e1f6c277a3122063cf5be9e150587152077d73c34d179a88e"
Jul 01 18:30:20.966191 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:30:20.966170    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f931d1246c444b4e1f6c277a3122063cf5be9e150587152077d73c34d179a88e\": container with ID starting with f931d1246c444b4e1f6c277a3122063cf5be9e150587152077d73c34d179a88e not found: ID does not exist" containerID="f931d1246c444b4e1f6c277a3122063cf5be9e150587152077d73c34d179a88e"
Jul 01 18:30:20.966258 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.966204    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f931d1246c444b4e1f6c277a3122063cf5be9e150587152077d73c34d179a88e"} err="failed to get container status \"f931d1246c444b4e1f6c277a3122063cf5be9e150587152077d73c34d179a88e\": rpc error: code = NotFound desc = could not find container \"f931d1246c444b4e1f6c277a3122063cf5be9e150587152077d73c34d179a88e\": container with ID starting with f931d1246c444b4e1f6c277a3122063cf5be9e150587152077d73c34d179a88e not found: ID does not exist"
Jul 01 18:30:20.966258 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.966224    2578 scope.go:117] "RemoveContainer" containerID="41cb51740db21fa8a7677ad38358ef87f3e66684bd801549a1e517adc4f51f93"
Jul 01 18:30:20.966505 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:30:20.966487    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41cb51740db21fa8a7677ad38358ef87f3e66684bd801549a1e517adc4f51f93\": container with ID starting with 41cb51740db21fa8a7677ad38358ef87f3e66684bd801549a1e517adc4f51f93 not found: ID does not exist" containerID="41cb51740db21fa8a7677ad38358ef87f3e66684bd801549a1e517adc4f51f93"
Jul 01 18:30:20.966566 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.966511    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cb51740db21fa8a7677ad38358ef87f3e66684bd801549a1e517adc4f51f93"} err="failed to get container status \"41cb51740db21fa8a7677ad38358ef87f3e66684bd801549a1e517adc4f51f93\": rpc error: code = NotFound desc = could not find container \"41cb51740db21fa8a7677ad38358ef87f3e66684bd801549a1e517adc4f51f93\": container with ID starting with 41cb51740db21fa8a7677ad38358ef87f3e66684bd801549a1e517adc4f51f93 not found: ID does not exist"
Jul 01 18:30:20.969147 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.969122    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"]
Jul 01 18:30:20.975279 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:20.975248    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2588a-predictor-75c88dcf4c-qp2m8"]
Jul 01 18:30:22.820596 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:22.820557    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" path="/var/lib/kubelet/pods/6d05b78c-7dca-4f07-b37a-007729f90494/volumes"
Jul 01 18:30:24.948166 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:24.948134    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:30:24.948744 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:24.948716    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 18:30:34.949078 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:34.949035    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 18:30:44.948712 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:44.948671    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 18:30:54.949239 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:30:54.949193    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 18:31:04.949157 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:04.949125    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:31:12.830501 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.830467    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"]
Jul 01 18:31:12.830972 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.830737    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" containerName="kube-rbac-proxy"
Jul 01 18:31:12.830972 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.830747    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" containerName="kube-rbac-proxy"
Jul 01 18:31:12.830972 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.830760    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" containerName="kserve-container"
Jul 01 18:31:12.830972 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.830766    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" containerName="kserve-container"
Jul 01 18:31:12.830972 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.830813    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" containerName="kserve-container"
Jul 01 18:31:12.830972 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.830820    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d05b78c-7dca-4f07-b37a-007729f90494" containerName="kube-rbac-proxy"
Jul 01 18:31:12.833830 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.833814    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:31:12.836689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.836661    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9b590-predictor-serving-cert\""
Jul 01 18:31:12.836836 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.836707    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9b590-kube-rbac-proxy-sar-config\""
Jul 01 18:31:12.842581 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.842556    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"]
Jul 01 18:31:12.868424 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.868393    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"]
Jul 01 18:31:12.868640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.868614    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kserve-container" containerID="cri-o://2182d49e661599e7aca96c4a22d62225fedf9d74b762e4ad4d7a0c72448d7fa8" gracePeriod=30
Jul 01 18:31:12.868703 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.868668    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kube-rbac-proxy" containerID="cri-o://f693b049141f4e67eead2066feeb4a03f00969c92b0d99a61990b97dbc19299d" gracePeriod=30
Jul 01 18:31:12.936727 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.936687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18bcbb28-c219-45dc-b867-abcf7f139ec3-proxy-tls\") pod \"error-404-isvc-9b590-predictor-954bbfb5c-65n5p\" (UID: \"18bcbb28-c219-45dc-b867-abcf7f139ec3\") " pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:31:12.936900 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.936745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18bcbb28-c219-45dc-b867-abcf7f139ec3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9b590-predictor-954bbfb5c-65n5p\" (UID: \"18bcbb28-c219-45dc-b867-abcf7f139ec3\") " pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:31:12.936900 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:12.936829    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfgfm\" (UniqueName: \"kubernetes.io/projected/18bcbb28-c219-45dc-b867-abcf7f139ec3-kube-api-access-nfgfm\") pod \"error-404-isvc-9b590-predictor-954bbfb5c-65n5p\" (UID: \"18bcbb28-c219-45dc-b867-abcf7f139ec3\") " pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:31:13.037377 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:13.037332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18bcbb28-c219-45dc-b867-abcf7f139ec3-proxy-tls\") pod \"error-404-isvc-9b590-predictor-954bbfb5c-65n5p\" (UID: \"18bcbb28-c219-45dc-b867-abcf7f139ec3\") " pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:31:13.037572 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:13.037390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18bcbb28-c219-45dc-b867-abcf7f139ec3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9b590-predictor-954bbfb5c-65n5p\" (UID: \"18bcbb28-c219-45dc-b867-abcf7f139ec3\") " pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:31:13.037572 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:13.037416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfgfm\" (UniqueName: \"kubernetes.io/projected/18bcbb28-c219-45dc-b867-abcf7f139ec3-kube-api-access-nfgfm\") pod \"error-404-isvc-9b590-predictor-954bbfb5c-65n5p\" (UID: \"18bcbb28-c219-45dc-b867-abcf7f139ec3\") " pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:31:13.038088 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:13.038056    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18bcbb28-c219-45dc-b867-abcf7f139ec3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9b590-predictor-954bbfb5c-65n5p\" (UID: \"18bcbb28-c219-45dc-b867-abcf7f139ec3\") " pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:31:13.040498 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:13.040470    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18bcbb28-c219-45dc-b867-abcf7f139ec3-proxy-tls\") pod \"error-404-isvc-9b590-predictor-954bbfb5c-65n5p\" (UID: \"18bcbb28-c219-45dc-b867-abcf7f139ec3\") " pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:31:13.045390 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:13.045364    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfgfm\" (UniqueName: \"kubernetes.io/projected/18bcbb28-c219-45dc-b867-abcf7f139ec3-kube-api-access-nfgfm\") pod \"error-404-isvc-9b590-predictor-954bbfb5c-65n5p\" (UID: \"18bcbb28-c219-45dc-b867-abcf7f139ec3\") " pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:31:13.145489 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:13.145391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:31:13.156598 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:13.156567    2578 generic.go:356] "Generic (PLEG): container finished" podID="6727f26d-7a38-47ac-95b0-74d409638145" containerID="f693b049141f4e67eead2066feeb4a03f00969c92b0d99a61990b97dbc19299d" exitCode=2
Jul 01 18:31:13.156729 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:13.156640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" event={"ID":"6727f26d-7a38-47ac-95b0-74d409638145","Type":"ContainerDied","Data":"f693b049141f4e67eead2066feeb4a03f00969c92b0d99a61990b97dbc19299d"}
Jul 01 18:31:13.278232 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:13.278204    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"]
Jul 01 18:31:13.280874 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:31:13.280845    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bcbb28_c219_45dc_b867_abcf7f139ec3.slice/crio-23de066de2440b498e299d9c5dc5a394f04e031f863392079b60de654e482e2e WatchSource:0}: Error finding container 23de066de2440b498e299d9c5dc5a394f04e031f863392079b60de654e482e2e: Status 404 returned error can't find the container with id 23de066de2440b498e299d9c5dc5a394f04e031f863392079b60de654e482e2e
Jul 01 18:31:14.163148 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:14.163107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" event={"ID":"18bcbb28-c219-45dc-b867-abcf7f139ec3","Type":"ContainerStarted","Data":"0ad8de765ac0c5bb56a62495cf105e2fa6d82533a1f8e92ca1e3afeadb4064ac"}
Jul 01 18:31:14.163589 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:14.163296    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:31:14.163589 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:14.163339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" event={"ID":"18bcbb28-c219-45dc-b867-abcf7f139ec3","Type":"ContainerStarted","Data":"a4dc96d35e4274d49d912bc90a264b5f3a5446b2bb4624ef7a36dbfd4d258f01"}
Jul 01 18:31:14.163589 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:14.163364    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" event={"ID":"18bcbb28-c219-45dc-b867-abcf7f139ec3","Type":"ContainerStarted","Data":"23de066de2440b498e299d9c5dc5a394f04e031f863392079b60de654e482e2e"}
Jul 01 18:31:14.181345 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:14.181289    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" podStartSLOduration=2.181274027 podStartE2EDuration="2.181274027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:31:14.180291866 +0000 UTC m=+1495.747576876" watchObservedRunningTime="2026-07-01 18:31:14.181274027 +0000 UTC m=+1495.748559006"
Jul 01 18:31:15.166662 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:15.166620    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:31:15.168018 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:15.167974    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 18:31:16.126164 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.126116    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 01 18:31:16.132366 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.132332    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 01 18:31:16.171562 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.171526    2578 generic.go:356] "Generic (PLEG): container finished" podID="6727f26d-7a38-47ac-95b0-74d409638145" containerID="2182d49e661599e7aca96c4a22d62225fedf9d74b762e4ad4d7a0c72448d7fa8" exitCode=0
Jul 01 18:31:16.171986 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.171593    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" event={"ID":"6727f26d-7a38-47ac-95b0-74d409638145","Type":"ContainerDied","Data":"2182d49e661599e7aca96c4a22d62225fedf9d74b762e4ad4d7a0c72448d7fa8"}
Jul 01 18:31:16.172059 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.172022    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 18:31:16.221791 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.221763    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:31:16.261196 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.261155    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnggd\" (UniqueName: \"kubernetes.io/projected/6727f26d-7a38-47ac-95b0-74d409638145-kube-api-access-jnggd\") pod \"6727f26d-7a38-47ac-95b0-74d409638145\" (UID: \"6727f26d-7a38-47ac-95b0-74d409638145\") "
Jul 01 18:31:16.261362 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.261215    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6727f26d-7a38-47ac-95b0-74d409638145-proxy-tls\") pod \"6727f26d-7a38-47ac-95b0-74d409638145\" (UID: \"6727f26d-7a38-47ac-95b0-74d409638145\") "
Jul 01 18:31:16.261362 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.261248    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6727f26d-7a38-47ac-95b0-74d409638145-kube-rbac-proxy-sar-config\") pod \"6727f26d-7a38-47ac-95b0-74d409638145\" (UID: \"6727f26d-7a38-47ac-95b0-74d409638145\") "
Jul 01 18:31:16.261703 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.261663    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6727f26d-7a38-47ac-95b0-74d409638145-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6727f26d-7a38-47ac-95b0-74d409638145" (UID: "6727f26d-7a38-47ac-95b0-74d409638145"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:31:16.263435 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.263408    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6727f26d-7a38-47ac-95b0-74d409638145-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6727f26d-7a38-47ac-95b0-74d409638145" (UID: "6727f26d-7a38-47ac-95b0-74d409638145"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:31:16.263530 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.263480    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6727f26d-7a38-47ac-95b0-74d409638145-kube-api-access-jnggd" (OuterVolumeSpecName: "kube-api-access-jnggd") pod "6727f26d-7a38-47ac-95b0-74d409638145" (UID: "6727f26d-7a38-47ac-95b0-74d409638145"). InnerVolumeSpecName "kube-api-access-jnggd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:31:16.362666 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.362576    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnggd\" (UniqueName: \"kubernetes.io/projected/6727f26d-7a38-47ac-95b0-74d409638145-kube-api-access-jnggd\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:31:16.362666 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.362607    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6727f26d-7a38-47ac-95b0-74d409638145-proxy-tls\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:31:16.362666 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:16.362618    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6727f26d-7a38-47ac-95b0-74d409638145-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:31:17.180387 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:17.180292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2" event={"ID":"6727f26d-7a38-47ac-95b0-74d409638145","Type":"ContainerDied","Data":"5344a18dc21b32136158e0d600dde1b3c7d9a3762073d13b1ed8466a40ba02ea"}
Jul 01 18:31:17.180387 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:17.180318    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"
Jul 01 18:31:17.180387 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:17.180368    2578 scope.go:117] "RemoveContainer" containerID="f693b049141f4e67eead2066feeb4a03f00969c92b0d99a61990b97dbc19299d"
Jul 01 18:31:17.188462 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:17.188441    2578 scope.go:117] "RemoveContainer" containerID="2182d49e661599e7aca96c4a22d62225fedf9d74b762e4ad4d7a0c72448d7fa8"
Jul 01 18:31:17.198140 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:17.198115    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"]
Jul 01 18:31:17.201706 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:17.201681    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5c10-predictor-dd55b8f87-dncv2"]
Jul 01 18:31:18.820072 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:18.820039    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6727f26d-7a38-47ac-95b0-74d409638145" path="/var/lib/kubelet/pods/6727f26d-7a38-47ac-95b0-74d409638145/volumes"
Jul 01 18:31:18.846464 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:18.846436    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:31:18.848105 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:18.848083    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:31:21.176189 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:21.176156    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:31:21.176673 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:21.176641    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 18:31:27.052143 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.052105    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"]
Jul 01 18:31:27.052637 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.052351    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kserve-container" containerID="cri-o://905f32789ede38d086b051191c7d231567b1c9549cfbd7c0c93960c75d284e01" gracePeriod=30
Jul 01 18:31:27.052637 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.052396    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kube-rbac-proxy" containerID="cri-o://15d1fc73e81c090beb0cf2b4ff3141ce7d6e3033581d031d1c2e95a57d035cc9" gracePeriod=30
Jul 01 18:31:27.224113 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.224074    2578 generic.go:356] "Generic (PLEG): container finished" podID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerID="15d1fc73e81c090beb0cf2b4ff3141ce7d6e3033581d031d1c2e95a57d035cc9" exitCode=2
Jul 01 18:31:27.224269 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.224150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" event={"ID":"48dac5a4-feb9-4052-9d7d-120cb2555d0e","Type":"ContainerDied","Data":"15d1fc73e81c090beb0cf2b4ff3141ce7d6e3033581d031d1c2e95a57d035cc9"}
Jul 01 18:31:27.228563 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.228528    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"]
Jul 01 18:31:27.228937 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.228920    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kserve-container"
Jul 01 18:31:27.229012 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.228949    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kserve-container"
Jul 01 18:31:27.229062 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.228986    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kube-rbac-proxy"
Jul 01 18:31:27.229062 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.229022    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kube-rbac-proxy"
Jul 01 18:31:27.229124 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.229103    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kube-rbac-proxy"
Jul 01 18:31:27.229124 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.229115    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6727f26d-7a38-47ac-95b0-74d409638145" containerName="kserve-container"
Jul 01 18:31:27.232597 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.232572    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:31:27.235488 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.235464    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c1eb4-predictor-serving-cert\""
Jul 01 18:31:27.235623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.235465    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c1eb4-kube-rbac-proxy-sar-config\""
Jul 01 18:31:27.243388 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.243364    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"]
Jul 01 18:31:27.350415 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.350309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5fd4\" (UniqueName: \"kubernetes.io/projected/d3410be8-a262-4434-beab-ca64a7d2774f-kube-api-access-t5fd4\") pod \"error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj\" (UID: \"d3410be8-a262-4434-beab-ca64a7d2774f\") " pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:31:27.350415 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.350385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3410be8-a262-4434-beab-ca64a7d2774f-proxy-tls\") pod \"error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj\" (UID: \"d3410be8-a262-4434-beab-ca64a7d2774f\") " pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:31:27.350621 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.350416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3410be8-a262-4434-beab-ca64a7d2774f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj\" (UID: \"d3410be8-a262-4434-beab-ca64a7d2774f\") " pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:31:27.451632 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.451587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5fd4\" (UniqueName: \"kubernetes.io/projected/d3410be8-a262-4434-beab-ca64a7d2774f-kube-api-access-t5fd4\") pod \"error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj\" (UID: \"d3410be8-a262-4434-beab-ca64a7d2774f\") " pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:31:27.451632 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.451635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3410be8-a262-4434-beab-ca64a7d2774f-proxy-tls\") pod \"error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj\" (UID: \"d3410be8-a262-4434-beab-ca64a7d2774f\") " pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:31:27.451900 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.451654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3410be8-a262-4434-beab-ca64a7d2774f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj\" (UID: \"d3410be8-a262-4434-beab-ca64a7d2774f\") " pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:31:27.452384 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.452364    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3410be8-a262-4434-beab-ca64a7d2774f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj\" (UID: \"d3410be8-a262-4434-beab-ca64a7d2774f\") " pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:31:27.454236 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.454214    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3410be8-a262-4434-beab-ca64a7d2774f-proxy-tls\") pod \"error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj\" (UID: \"d3410be8-a262-4434-beab-ca64a7d2774f\") " pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:31:27.476431 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.476396    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5fd4\" (UniqueName: \"kubernetes.io/projected/d3410be8-a262-4434-beab-ca64a7d2774f-kube-api-access-t5fd4\") pod \"error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj\" (UID: \"d3410be8-a262-4434-beab-ca64a7d2774f\") " pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:31:27.544286 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.544246    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:31:27.679952 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:27.679897    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"]
Jul 01 18:31:27.683952 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:31:27.683919    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3410be8_a262_4434_beab_ca64a7d2774f.slice/crio-7ee42547c4387316c1c028ae1743e66fceeabe7d68869342df1a8d92b796ad28 WatchSource:0}: Error finding container 7ee42547c4387316c1c028ae1743e66fceeabe7d68869342df1a8d92b796ad28: Status 404 returned error can't find the container with id 7ee42547c4387316c1c028ae1743e66fceeabe7d68869342df1a8d92b796ad28
Jul 01 18:31:28.230458 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:28.230414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" event={"ID":"d3410be8-a262-4434-beab-ca64a7d2774f","Type":"ContainerStarted","Data":"7fda7609458e6aedead31833856dab64abd4bded15127ce27b8148a61e62f788"}
Jul 01 18:31:28.230915 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:28.230532    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:31:28.230915 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:28.230569    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" event={"ID":"d3410be8-a262-4434-beab-ca64a7d2774f","Type":"ContainerStarted","Data":"c11dbc9dfcb2815404820823563579d13e56abc148d845235b08ffd73a6d1468"}
Jul 01 18:31:28.230915 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:28.230603    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" event={"ID":"d3410be8-a262-4434-beab-ca64a7d2774f","Type":"ContainerStarted","Data":"7ee42547c4387316c1c028ae1743e66fceeabe7d68869342df1a8d92b796ad28"}
Jul 01 18:31:28.230915 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:28.230637    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:31:28.231936 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:28.231902    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 18:31:28.248564 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:28.248515    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" podStartSLOduration=1.248497905 podStartE2EDuration="1.248497905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:31:28.24722936 +0000 UTC m=+1509.814514340" watchObservedRunningTime="2026-07-01 18:31:28.248497905 +0000 UTC m=+1509.815782886"
Jul 01 18:31:29.235485 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:29.235444    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 18:31:29.943488 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:29.943445    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 01 18:31:30.798210 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:30.798184    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:31:30.878796 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:30.878758    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps2xl\" (UniqueName: \"kubernetes.io/projected/48dac5a4-feb9-4052-9d7d-120cb2555d0e-kube-api-access-ps2xl\") pod \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\" (UID: \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\") "
Jul 01 18:31:30.879016 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:30.878814    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48dac5a4-feb9-4052-9d7d-120cb2555d0e-kube-rbac-proxy-sar-config\") pod \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\" (UID: \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\") "
Jul 01 18:31:30.879016 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:30.878913    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48dac5a4-feb9-4052-9d7d-120cb2555d0e-proxy-tls\") pod \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\" (UID: \"48dac5a4-feb9-4052-9d7d-120cb2555d0e\") "
Jul 01 18:31:30.879216 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:30.879188    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48dac5a4-feb9-4052-9d7d-120cb2555d0e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "48dac5a4-feb9-4052-9d7d-120cb2555d0e" (UID: "48dac5a4-feb9-4052-9d7d-120cb2555d0e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:31:30.880960 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:30.880899    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dac5a4-feb9-4052-9d7d-120cb2555d0e-kube-api-access-ps2xl" (OuterVolumeSpecName: "kube-api-access-ps2xl") pod "48dac5a4-feb9-4052-9d7d-120cb2555d0e" (UID: "48dac5a4-feb9-4052-9d7d-120cb2555d0e"). InnerVolumeSpecName "kube-api-access-ps2xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:31:30.881046 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:30.880953    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48dac5a4-feb9-4052-9d7d-120cb2555d0e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "48dac5a4-feb9-4052-9d7d-120cb2555d0e" (UID: "48dac5a4-feb9-4052-9d7d-120cb2555d0e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:31:30.979810 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:30.979764    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48dac5a4-feb9-4052-9d7d-120cb2555d0e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:31:30.979810 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:30.979800    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48dac5a4-feb9-4052-9d7d-120cb2555d0e-proxy-tls\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:31:30.979810 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:30.979811    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps2xl\" (UniqueName: \"kubernetes.io/projected/48dac5a4-feb9-4052-9d7d-120cb2555d0e-kube-api-access-ps2xl\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:31:31.177093 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:31.176981    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 18:31:31.244010 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:31.243959    2578 generic.go:356] "Generic (PLEG): container finished" podID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerID="905f32789ede38d086b051191c7d231567b1c9549cfbd7c0c93960c75d284e01" exitCode=0
Jul 01 18:31:31.244010 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:31.244013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" event={"ID":"48dac5a4-feb9-4052-9d7d-120cb2555d0e","Type":"ContainerDied","Data":"905f32789ede38d086b051191c7d231567b1c9549cfbd7c0c93960c75d284e01"}
Jul 01 18:31:31.244215 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:31.244034    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"
Jul 01 18:31:31.244215 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:31.244060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd" event={"ID":"48dac5a4-feb9-4052-9d7d-120cb2555d0e","Type":"ContainerDied","Data":"8e8b446a6e88b47fe591730df3b7a62a078b7c15c043e7c3869215ff16c25d52"}
Jul 01 18:31:31.244215 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:31.244087    2578 scope.go:117] "RemoveContainer" containerID="15d1fc73e81c090beb0cf2b4ff3141ce7d6e3033581d031d1c2e95a57d035cc9"
Jul 01 18:31:31.252575 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:31.252554    2578 scope.go:117] "RemoveContainer" containerID="905f32789ede38d086b051191c7d231567b1c9549cfbd7c0c93960c75d284e01"
Jul 01 18:31:31.260031 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:31.260006    2578 scope.go:117] "RemoveContainer" containerID="15d1fc73e81c090beb0cf2b4ff3141ce7d6e3033581d031d1c2e95a57d035cc9"
Jul 01 18:31:31.260328 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:31:31.260308    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d1fc73e81c090beb0cf2b4ff3141ce7d6e3033581d031d1c2e95a57d035cc9\": container with ID starting with 15d1fc73e81c090beb0cf2b4ff3141ce7d6e3033581d031d1c2e95a57d035cc9 not found: ID does not exist" containerID="15d1fc73e81c090beb0cf2b4ff3141ce7d6e3033581d031d1c2e95a57d035cc9"
Jul 01 18:31:31.260385 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:31.260337    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d1fc73e81c090beb0cf2b4ff3141ce7d6e3033581d031d1c2e95a57d035cc9"} err="failed to get container status \"15d1fc73e81c090beb0cf2b4ff3141ce7d6e3033581d031d1c2e95a57d035cc9\": rpc error: code = NotFound desc = could not find container \"15d1fc73e81c090beb0cf2b4ff3141ce7d6e3033581d031d1c2e95a57d035cc9\": container with ID starting with 15d1fc73e81c090beb0cf2b4ff3141ce7d6e3033581d031d1c2e95a57d035cc9 not found: ID does not exist"
Jul 01 18:31:31.260385 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:31.260355    2578 scope.go:117] "RemoveContainer" containerID="905f32789ede38d086b051191c7d231567b1c9549cfbd7c0c93960c75d284e01"
Jul 01 18:31:31.260606 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:31:31.260588    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905f32789ede38d086b051191c7d231567b1c9549cfbd7c0c93960c75d284e01\": container with ID starting with 905f32789ede38d086b051191c7d231567b1c9549cfbd7c0c93960c75d284e01 not found: ID does not exist" containerID="905f32789ede38d086b051191c7d231567b1c9549cfbd7c0c93960c75d284e01"
Jul 01 18:31:31.260659 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:31.260615    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905f32789ede38d086b051191c7d231567b1c9549cfbd7c0c93960c75d284e01"} err="failed to get container status \"905f32789ede38d086b051191c7d231567b1c9549cfbd7c0c93960c75d284e01\": rpc error: code = NotFound desc = could not find container \"905f32789ede38d086b051191c7d231567b1c9549cfbd7c0c93960c75d284e01\": container with ID starting with 905f32789ede38d086b051191c7d231567b1c9549cfbd7c0c93960c75d284e01 not found: ID does not exist"
Jul 01 18:31:31.264738 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:31.264712    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"]
Jul 01 18:31:31.268923 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:31.268896    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0d3ed-predictor-6d6c58bcc5-545nd"]
Jul 01 18:31:32.820115 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:32.820083    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" path="/var/lib/kubelet/pods/48dac5a4-feb9-4052-9d7d-120cb2555d0e/volumes"
Jul 01 18:31:34.240119 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:34.240090    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:31:34.240597 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:34.240572    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 18:31:41.177400 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:41.177313    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 18:31:44.241312 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:44.241273    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 18:31:51.177240 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:51.177198    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 18:31:54.241198 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:31:54.241156    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 18:32:01.178109 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:01.178073    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:32:04.241146 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:04.241103    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 18:32:14.240612 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:14.240572    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 18:32:22.922973 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:22.922935    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"]
Jul 01 18:32:22.923512 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:22.923186    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kserve-container" containerID="cri-o://a4dc96d35e4274d49d912bc90a264b5f3a5446b2bb4624ef7a36dbfd4d258f01" gracePeriod=30
Jul 01 18:32:22.923512 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:22.923214    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kube-rbac-proxy" containerID="cri-o://0ad8de765ac0c5bb56a62495cf105e2fa6d82533a1f8e92ca1e3afeadb4064ac" gracePeriod=30
Jul 01 18:32:23.077764 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.077726    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"]
Jul 01 18:32:23.078054 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.078041    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kserve-container"
Jul 01 18:32:23.078104 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.078056    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kserve-container"
Jul 01 18:32:23.078104 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.078075    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kube-rbac-proxy"
Jul 01 18:32:23.078104 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.078081    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kube-rbac-proxy"
Jul 01 18:32:23.078191 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.078126    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kube-rbac-proxy"
Jul 01 18:32:23.078191 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.078134    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="48dac5a4-feb9-4052-9d7d-120cb2555d0e" containerName="kserve-container"
Jul 01 18:32:23.081065 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.081043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:32:23.083486 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.083460    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-293b3-kube-rbac-proxy-sar-config\""
Jul 01 18:32:23.083591 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.083460    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-293b3-predictor-serving-cert\""
Jul 01 18:32:23.090944 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.090912    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"]
Jul 01 18:32:23.170925 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.170887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cac7e284-aea5-4817-b2ea-927dbc78868a-proxy-tls\") pod \"error-404-isvc-293b3-predictor-864bd98777-svbdl\" (UID: \"cac7e284-aea5-4817-b2ea-927dbc78868a\") " pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:32:23.171137 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.170947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66xq\" (UniqueName: \"kubernetes.io/projected/cac7e284-aea5-4817-b2ea-927dbc78868a-kube-api-access-p66xq\") pod \"error-404-isvc-293b3-predictor-864bd98777-svbdl\" (UID: \"cac7e284-aea5-4817-b2ea-927dbc78868a\") " pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:32:23.171137 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.171030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cac7e284-aea5-4817-b2ea-927dbc78868a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-293b3-predictor-864bd98777-svbdl\" (UID: \"cac7e284-aea5-4817-b2ea-927dbc78868a\") " pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:32:23.271518 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.271412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66xq\" (UniqueName: \"kubernetes.io/projected/cac7e284-aea5-4817-b2ea-927dbc78868a-kube-api-access-p66xq\") pod \"error-404-isvc-293b3-predictor-864bd98777-svbdl\" (UID: \"cac7e284-aea5-4817-b2ea-927dbc78868a\") " pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:32:23.271518 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.271452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cac7e284-aea5-4817-b2ea-927dbc78868a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-293b3-predictor-864bd98777-svbdl\" (UID: \"cac7e284-aea5-4817-b2ea-927dbc78868a\") " pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:32:23.271763 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.271579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cac7e284-aea5-4817-b2ea-927dbc78868a-proxy-tls\") pod \"error-404-isvc-293b3-predictor-864bd98777-svbdl\" (UID: \"cac7e284-aea5-4817-b2ea-927dbc78868a\") " pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:32:23.272183 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.272165    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cac7e284-aea5-4817-b2ea-927dbc78868a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-293b3-predictor-864bd98777-svbdl\" (UID: \"cac7e284-aea5-4817-b2ea-927dbc78868a\") " pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:32:23.274193 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.274173    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cac7e284-aea5-4817-b2ea-927dbc78868a-proxy-tls\") pod \"error-404-isvc-293b3-predictor-864bd98777-svbdl\" (UID: \"cac7e284-aea5-4817-b2ea-927dbc78868a\") " pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:32:23.280322 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.280292    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66xq\" (UniqueName: \"kubernetes.io/projected/cac7e284-aea5-4817-b2ea-927dbc78868a-kube-api-access-p66xq\") pod \"error-404-isvc-293b3-predictor-864bd98777-svbdl\" (UID: \"cac7e284-aea5-4817-b2ea-927dbc78868a\") " pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:32:23.392575 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.392531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:32:23.437729 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.437688    2578 generic.go:356] "Generic (PLEG): container finished" podID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerID="0ad8de765ac0c5bb56a62495cf105e2fa6d82533a1f8e92ca1e3afeadb4064ac" exitCode=2
Jul 01 18:32:23.437894 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.437757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" event={"ID":"18bcbb28-c219-45dc-b867-abcf7f139ec3","Type":"ContainerDied","Data":"0ad8de765ac0c5bb56a62495cf105e2fa6d82533a1f8e92ca1e3afeadb4064ac"}
Jul 01 18:32:23.524733 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:23.524577    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"]
Jul 01 18:32:23.527303 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:32:23.527271    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac7e284_aea5_4817_b2ea_927dbc78868a.slice/crio-f9557b8fceecb34375525ab5127b237343b39baea718e76eb3615ca0336aab23 WatchSource:0}: Error finding container f9557b8fceecb34375525ab5127b237343b39baea718e76eb3615ca0336aab23: Status 404 returned error can't find the container with id f9557b8fceecb34375525ab5127b237343b39baea718e76eb3615ca0336aab23
Jul 01 18:32:24.242042 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:24.241987    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:32:24.442827 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:24.442785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" event={"ID":"cac7e284-aea5-4817-b2ea-927dbc78868a","Type":"ContainerStarted","Data":"1c0540b136417949d34c8f3d3dc51413fe681deca84b49258526c1a79c56c17e"}
Jul 01 18:32:24.442976 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:24.442850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" event={"ID":"cac7e284-aea5-4817-b2ea-927dbc78868a","Type":"ContainerStarted","Data":"ed4cb48ea3c0c7bb1345ca4344b8ab1c51bc5a1df95e7a267336aeedbae941f2"}
Jul 01 18:32:24.442976 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:24.442899    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:32:24.442976 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:24.442918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" event={"ID":"cac7e284-aea5-4817-b2ea-927dbc78868a","Type":"ContainerStarted","Data":"f9557b8fceecb34375525ab5127b237343b39baea718e76eb3615ca0336aab23"}
Jul 01 18:32:25.447867 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:25.447829    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:32:25.449170 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:25.449134    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 18:32:26.172786 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.172747    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 01 18:32:26.387181 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.387111    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:32:26.407429 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.407380    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" podStartSLOduration=3.407362776 podStartE2EDuration="3.407362776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:32:24.459983015 +0000 UTC m=+1566.027267996" watchObservedRunningTime="2026-07-01 18:32:26.407362776 +0000 UTC m=+1567.974647773"
Jul 01 18:32:26.453441 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.453405    2578 generic.go:356] "Generic (PLEG): container finished" podID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerID="a4dc96d35e4274d49d912bc90a264b5f3a5446b2bb4624ef7a36dbfd4d258f01" exitCode=0
Jul 01 18:32:26.453867 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.453455    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"
Jul 01 18:32:26.453867 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.453494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" event={"ID":"18bcbb28-c219-45dc-b867-abcf7f139ec3","Type":"ContainerDied","Data":"a4dc96d35e4274d49d912bc90a264b5f3a5446b2bb4624ef7a36dbfd4d258f01"}
Jul 01 18:32:26.453867 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.453554    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p" event={"ID":"18bcbb28-c219-45dc-b867-abcf7f139ec3","Type":"ContainerDied","Data":"23de066de2440b498e299d9c5dc5a394f04e031f863392079b60de654e482e2e"}
Jul 01 18:32:26.453867 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.453583    2578 scope.go:117] "RemoveContainer" containerID="0ad8de765ac0c5bb56a62495cf105e2fa6d82533a1f8e92ca1e3afeadb4064ac"
Jul 01 18:32:26.454101 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.453978    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 18:32:26.461715 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.461698    2578 scope.go:117] "RemoveContainer" containerID="a4dc96d35e4274d49d912bc90a264b5f3a5446b2bb4624ef7a36dbfd4d258f01"
Jul 01 18:32:26.469263 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.469243    2578 scope.go:117] "RemoveContainer" containerID="0ad8de765ac0c5bb56a62495cf105e2fa6d82533a1f8e92ca1e3afeadb4064ac"
Jul 01 18:32:26.469557 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:32:26.469534    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad8de765ac0c5bb56a62495cf105e2fa6d82533a1f8e92ca1e3afeadb4064ac\": container with ID starting with 0ad8de765ac0c5bb56a62495cf105e2fa6d82533a1f8e92ca1e3afeadb4064ac not found: ID does not exist" containerID="0ad8de765ac0c5bb56a62495cf105e2fa6d82533a1f8e92ca1e3afeadb4064ac"
Jul 01 18:32:26.469603 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.469569    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad8de765ac0c5bb56a62495cf105e2fa6d82533a1f8e92ca1e3afeadb4064ac"} err="failed to get container status \"0ad8de765ac0c5bb56a62495cf105e2fa6d82533a1f8e92ca1e3afeadb4064ac\": rpc error: code = NotFound desc = could not find container \"0ad8de765ac0c5bb56a62495cf105e2fa6d82533a1f8e92ca1e3afeadb4064ac\": container with ID starting with 0ad8de765ac0c5bb56a62495cf105e2fa6d82533a1f8e92ca1e3afeadb4064ac not found: ID does not exist"
Jul 01 18:32:26.469603 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.469588    2578 scope.go:117] "RemoveContainer" containerID="a4dc96d35e4274d49d912bc90a264b5f3a5446b2bb4624ef7a36dbfd4d258f01"
Jul 01 18:32:26.469847 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:32:26.469829    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4dc96d35e4274d49d912bc90a264b5f3a5446b2bb4624ef7a36dbfd4d258f01\": container with ID starting with a4dc96d35e4274d49d912bc90a264b5f3a5446b2bb4624ef7a36dbfd4d258f01 not found: ID does not exist" containerID="a4dc96d35e4274d49d912bc90a264b5f3a5446b2bb4624ef7a36dbfd4d258f01"
Jul 01 18:32:26.469888 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.469853    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4dc96d35e4274d49d912bc90a264b5f3a5446b2bb4624ef7a36dbfd4d258f01"} err="failed to get container status \"a4dc96d35e4274d49d912bc90a264b5f3a5446b2bb4624ef7a36dbfd4d258f01\": rpc error: code = NotFound desc = could not find container \"a4dc96d35e4274d49d912bc90a264b5f3a5446b2bb4624ef7a36dbfd4d258f01\": container with ID starting with a4dc96d35e4274d49d912bc90a264b5f3a5446b2bb4624ef7a36dbfd4d258f01 not found: ID does not exist"
Jul 01 18:32:26.503190 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.503156    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfgfm\" (UniqueName: \"kubernetes.io/projected/18bcbb28-c219-45dc-b867-abcf7f139ec3-kube-api-access-nfgfm\") pod \"18bcbb28-c219-45dc-b867-abcf7f139ec3\" (UID: \"18bcbb28-c219-45dc-b867-abcf7f139ec3\") "
Jul 01 18:32:26.503357 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.503230    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18bcbb28-c219-45dc-b867-abcf7f139ec3-kube-rbac-proxy-sar-config\") pod \"18bcbb28-c219-45dc-b867-abcf7f139ec3\" (UID: \"18bcbb28-c219-45dc-b867-abcf7f139ec3\") "
Jul 01 18:32:26.503357 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.503280    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18bcbb28-c219-45dc-b867-abcf7f139ec3-proxy-tls\") pod \"18bcbb28-c219-45dc-b867-abcf7f139ec3\" (UID: \"18bcbb28-c219-45dc-b867-abcf7f139ec3\") "
Jul 01 18:32:26.503650 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.503621    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18bcbb28-c219-45dc-b867-abcf7f139ec3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "18bcbb28-c219-45dc-b867-abcf7f139ec3" (UID: "18bcbb28-c219-45dc-b867-abcf7f139ec3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:32:26.505350 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.505323    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bcbb28-c219-45dc-b867-abcf7f139ec3-kube-api-access-nfgfm" (OuterVolumeSpecName: "kube-api-access-nfgfm") pod "18bcbb28-c219-45dc-b867-abcf7f139ec3" (UID: "18bcbb28-c219-45dc-b867-abcf7f139ec3"). InnerVolumeSpecName "kube-api-access-nfgfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:32:26.505428 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.505380    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18bcbb28-c219-45dc-b867-abcf7f139ec3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "18bcbb28-c219-45dc-b867-abcf7f139ec3" (UID: "18bcbb28-c219-45dc-b867-abcf7f139ec3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:32:26.604034 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.603967    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18bcbb28-c219-45dc-b867-abcf7f139ec3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:32:26.604034 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.604034    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18bcbb28-c219-45dc-b867-abcf7f139ec3-proxy-tls\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:32:26.604242 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.604051    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfgfm\" (UniqueName: \"kubernetes.io/projected/18bcbb28-c219-45dc-b867-abcf7f139ec3-kube-api-access-nfgfm\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:32:26.774710 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.774675    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"]
Jul 01 18:32:26.777868 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.777839    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9b590-predictor-954bbfb5c-65n5p"]
Jul 01 18:32:26.820839 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:26.820801    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" path="/var/lib/kubelet/pods/18bcbb28-c219-45dc-b867-abcf7f139ec3/volumes"
Jul 01 18:32:31.458857 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:31.458821    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:32:31.459471 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:31.459443    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 18:32:41.460122 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:41.460079    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 18:32:51.459804 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:32:51.459756    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 18:33:01.459802 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:33:01.459755    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 18:33:11.460729 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:33:11.460643    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:36:18.866506 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:36:18.866472    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:36:18.875217 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:36:18.875184    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:40:42.008308 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.008223    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"]
Jul 01 18:40:42.010666 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.008515    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kube-rbac-proxy"
Jul 01 18:40:42.010666 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.008527    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kube-rbac-proxy"
Jul 01 18:40:42.010666 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.008540    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kserve-container"
Jul 01 18:40:42.010666 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.008546    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kserve-container"
Jul 01 18:40:42.010666 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.008586    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kube-rbac-proxy"
Jul 01 18:40:42.010666 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.008596    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="18bcbb28-c219-45dc-b867-abcf7f139ec3" containerName="kserve-container"
Jul 01 18:40:42.011623 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.011605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:42.014361 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.014327    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fbcf2-predictor-serving-cert\""
Jul 01 18:40:42.014361 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.014356    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fbcf2-kube-rbac-proxy-sar-config\""
Jul 01 18:40:42.021290 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.021266    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"]
Jul 01 18:40:42.071578 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.071546    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"]
Jul 01 18:40:42.071798 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.071772    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kserve-container" containerID="cri-o://c11dbc9dfcb2815404820823563579d13e56abc148d845235b08ffd73a6d1468" gracePeriod=30
Jul 01 18:40:42.071879 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.071851    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kube-rbac-proxy" containerID="cri-o://7fda7609458e6aedead31833856dab64abd4bded15127ce27b8148a61e62f788" gracePeriod=30
Jul 01 18:40:42.128302 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.128260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vj7\" (UniqueName: \"kubernetes.io/projected/3f64d70e-353c-4fe5-bdbc-52e32677e226-kube-api-access-k5vj7\") pod \"error-404-isvc-fbcf2-predictor-748c594f6f-tlrls\" (UID: \"3f64d70e-353c-4fe5-bdbc-52e32677e226\") " pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:42.128466 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.128310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f64d70e-353c-4fe5-bdbc-52e32677e226-proxy-tls\") pod \"error-404-isvc-fbcf2-predictor-748c594f6f-tlrls\" (UID: \"3f64d70e-353c-4fe5-bdbc-52e32677e226\") " pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:42.128466 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.128413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f64d70e-353c-4fe5-bdbc-52e32677e226-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fbcf2-predictor-748c594f6f-tlrls\" (UID: \"3f64d70e-353c-4fe5-bdbc-52e32677e226\") " pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:42.229052 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.229012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f64d70e-353c-4fe5-bdbc-52e32677e226-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fbcf2-predictor-748c594f6f-tlrls\" (UID: \"3f64d70e-353c-4fe5-bdbc-52e32677e226\") " pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:42.229250 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.229184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vj7\" (UniqueName: \"kubernetes.io/projected/3f64d70e-353c-4fe5-bdbc-52e32677e226-kube-api-access-k5vj7\") pod \"error-404-isvc-fbcf2-predictor-748c594f6f-tlrls\" (UID: \"3f64d70e-353c-4fe5-bdbc-52e32677e226\") " pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:42.229250 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.229242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f64d70e-353c-4fe5-bdbc-52e32677e226-proxy-tls\") pod \"error-404-isvc-fbcf2-predictor-748c594f6f-tlrls\" (UID: \"3f64d70e-353c-4fe5-bdbc-52e32677e226\") " pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:42.229370 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:40:42.229355    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-serving-cert: secret "error-404-isvc-fbcf2-predictor-serving-cert" not found
Jul 01 18:40:42.229433 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:40:42.229424    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f64d70e-353c-4fe5-bdbc-52e32677e226-proxy-tls podName:3f64d70e-353c-4fe5-bdbc-52e32677e226 nodeName:}" failed. No retries permitted until 2026-07-01 18:40:42.729408526 +0000 UTC m=+2064.296693483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3f64d70e-353c-4fe5-bdbc-52e32677e226-proxy-tls") pod "error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" (UID: "3f64d70e-353c-4fe5-bdbc-52e32677e226") : secret "error-404-isvc-fbcf2-predictor-serving-cert" not found
Jul 01 18:40:42.229705 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.229686    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f64d70e-353c-4fe5-bdbc-52e32677e226-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fbcf2-predictor-748c594f6f-tlrls\" (UID: \"3f64d70e-353c-4fe5-bdbc-52e32677e226\") " pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:42.237795 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.237766    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vj7\" (UniqueName: \"kubernetes.io/projected/3f64d70e-353c-4fe5-bdbc-52e32677e226-kube-api-access-k5vj7\") pod \"error-404-isvc-fbcf2-predictor-748c594f6f-tlrls\" (UID: \"3f64d70e-353c-4fe5-bdbc-52e32677e226\") " pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:42.251869 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.251835    2578 generic.go:356] "Generic (PLEG): container finished" podID="d3410be8-a262-4434-beab-ca64a7d2774f" containerID="7fda7609458e6aedead31833856dab64abd4bded15127ce27b8148a61e62f788" exitCode=2
Jul 01 18:40:42.252041 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.251907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" event={"ID":"d3410be8-a262-4434-beab-ca64a7d2774f","Type":"ContainerDied","Data":"7fda7609458e6aedead31833856dab64abd4bded15127ce27b8148a61e62f788"}
Jul 01 18:40:42.733297 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.733253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f64d70e-353c-4fe5-bdbc-52e32677e226-proxy-tls\") pod \"error-404-isvc-fbcf2-predictor-748c594f6f-tlrls\" (UID: \"3f64d70e-353c-4fe5-bdbc-52e32677e226\") " pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:42.735811 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.735784    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f64d70e-353c-4fe5-bdbc-52e32677e226-proxy-tls\") pod \"error-404-isvc-fbcf2-predictor-748c594f6f-tlrls\" (UID: \"3f64d70e-353c-4fe5-bdbc-52e32677e226\") " pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:42.922603 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:42.922566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:43.054381 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:43.054346    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"]
Jul 01 18:40:43.065987 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:40:43.065953    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f64d70e_353c_4fe5_bdbc_52e32677e226.slice/crio-b5871673604453383eb3256c635e22f38162371f7596d62589192324541aa433 WatchSource:0}: Error finding container b5871673604453383eb3256c635e22f38162371f7596d62589192324541aa433: Status 404 returned error can't find the container with id b5871673604453383eb3256c635e22f38162371f7596d62589192324541aa433
Jul 01 18:40:43.257761 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:43.257674    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" event={"ID":"3f64d70e-353c-4fe5-bdbc-52e32677e226","Type":"ContainerStarted","Data":"6ea81fa88af1fc42dd82332d7abcbc8d91f7089cbb3411ec34da9335decd086a"}
Jul 01 18:40:43.257761 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:43.257734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" event={"ID":"3f64d70e-353c-4fe5-bdbc-52e32677e226","Type":"ContainerStarted","Data":"8e8e654a22da92257dd70e8989b5a34aeee736a30108e70fcb3c11b89b194228"}
Jul 01 18:40:43.257761 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:43.257754    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" event={"ID":"3f64d70e-353c-4fe5-bdbc-52e32677e226","Type":"ContainerStarted","Data":"b5871673604453383eb3256c635e22f38162371f7596d62589192324541aa433"}
Jul 01 18:40:43.257980 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:43.257806    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:43.275512 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:43.275461    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" podStartSLOduration=2.275445826 podStartE2EDuration="2.275445826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:40:43.27432895 +0000 UTC m=+2064.841613941" watchObservedRunningTime="2026-07-01 18:40:43.275445826 +0000 UTC m=+2064.842730804"
Jul 01 18:40:44.236422 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:44.236379    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 01 18:40:44.240756 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:44.240722    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 18:40:44.261222 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:44.261178    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:44.262480 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:44.262453    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 18:40:45.264694 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:45.264653    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 18:40:48.531178 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:48.531144    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:40:48.679396 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:48.679290    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3410be8-a262-4434-beab-ca64a7d2774f-kube-rbac-proxy-sar-config\") pod \"d3410be8-a262-4434-beab-ca64a7d2774f\" (UID: \"d3410be8-a262-4434-beab-ca64a7d2774f\") "
Jul 01 18:40:48.679396 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:48.679339    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5fd4\" (UniqueName: \"kubernetes.io/projected/d3410be8-a262-4434-beab-ca64a7d2774f-kube-api-access-t5fd4\") pod \"d3410be8-a262-4434-beab-ca64a7d2774f\" (UID: \"d3410be8-a262-4434-beab-ca64a7d2774f\") "
Jul 01 18:40:48.679396 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:48.679383    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3410be8-a262-4434-beab-ca64a7d2774f-proxy-tls\") pod \"d3410be8-a262-4434-beab-ca64a7d2774f\" (UID: \"d3410be8-a262-4434-beab-ca64a7d2774f\") "
Jul 01 18:40:48.679692 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:48.679667    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3410be8-a262-4434-beab-ca64a7d2774f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3410be8-a262-4434-beab-ca64a7d2774f" (UID: "d3410be8-a262-4434-beab-ca64a7d2774f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:48.681613 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:48.681579    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3410be8-a262-4434-beab-ca64a7d2774f-kube-api-access-t5fd4" (OuterVolumeSpecName: "kube-api-access-t5fd4") pod "d3410be8-a262-4434-beab-ca64a7d2774f" (UID: "d3410be8-a262-4434-beab-ca64a7d2774f"). InnerVolumeSpecName "kube-api-access-t5fd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:40:48.681613 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:48.681593    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3410be8-a262-4434-beab-ca64a7d2774f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3410be8-a262-4434-beab-ca64a7d2774f" (UID: "d3410be8-a262-4434-beab-ca64a7d2774f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:48.780343 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:48.780291    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3410be8-a262-4434-beab-ca64a7d2774f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:40:48.780343 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:48.780336    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5fd4\" (UniqueName: \"kubernetes.io/projected/d3410be8-a262-4434-beab-ca64a7d2774f-kube-api-access-t5fd4\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:40:48.780343 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:48.780347    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3410be8-a262-4434-beab-ca64a7d2774f-proxy-tls\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:40:49.281534 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:49.281494    2578 generic.go:356] "Generic (PLEG): container finished" podID="d3410be8-a262-4434-beab-ca64a7d2774f" containerID="c11dbc9dfcb2815404820823563579d13e56abc148d845235b08ffd73a6d1468" exitCode=0
Jul 01 18:40:49.281742 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:49.281553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" event={"ID":"d3410be8-a262-4434-beab-ca64a7d2774f","Type":"ContainerDied","Data":"c11dbc9dfcb2815404820823563579d13e56abc148d845235b08ffd73a6d1468"}
Jul 01 18:40:49.281742 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:49.281561    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"
Jul 01 18:40:49.281742 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:49.281602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj" event={"ID":"d3410be8-a262-4434-beab-ca64a7d2774f","Type":"ContainerDied","Data":"7ee42547c4387316c1c028ae1743e66fceeabe7d68869342df1a8d92b796ad28"}
Jul 01 18:40:49.281742 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:49.281628    2578 scope.go:117] "RemoveContainer" containerID="7fda7609458e6aedead31833856dab64abd4bded15127ce27b8148a61e62f788"
Jul 01 18:40:49.290331 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:49.290310    2578 scope.go:117] "RemoveContainer" containerID="c11dbc9dfcb2815404820823563579d13e56abc148d845235b08ffd73a6d1468"
Jul 01 18:40:49.298865 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:49.298842    2578 scope.go:117] "RemoveContainer" containerID="7fda7609458e6aedead31833856dab64abd4bded15127ce27b8148a61e62f788"
Jul 01 18:40:49.299231 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:40:49.299170    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fda7609458e6aedead31833856dab64abd4bded15127ce27b8148a61e62f788\": container with ID starting with 7fda7609458e6aedead31833856dab64abd4bded15127ce27b8148a61e62f788 not found: ID does not exist" containerID="7fda7609458e6aedead31833856dab64abd4bded15127ce27b8148a61e62f788"
Jul 01 18:40:49.299231 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:49.299206    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fda7609458e6aedead31833856dab64abd4bded15127ce27b8148a61e62f788"} err="failed to get container status \"7fda7609458e6aedead31833856dab64abd4bded15127ce27b8148a61e62f788\": rpc error: code = NotFound desc = could not find container \"7fda7609458e6aedead31833856dab64abd4bded15127ce27b8148a61e62f788\": container with ID starting with 7fda7609458e6aedead31833856dab64abd4bded15127ce27b8148a61e62f788 not found: ID does not exist"
Jul 01 18:40:49.299231 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:49.299234    2578 scope.go:117] "RemoveContainer" containerID="c11dbc9dfcb2815404820823563579d13e56abc148d845235b08ffd73a6d1468"
Jul 01 18:40:49.299584 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:40:49.299553    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11dbc9dfcb2815404820823563579d13e56abc148d845235b08ffd73a6d1468\": container with ID starting with c11dbc9dfcb2815404820823563579d13e56abc148d845235b08ffd73a6d1468 not found: ID does not exist" containerID="c11dbc9dfcb2815404820823563579d13e56abc148d845235b08ffd73a6d1468"
Jul 01 18:40:49.299675 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:49.299607    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11dbc9dfcb2815404820823563579d13e56abc148d845235b08ffd73a6d1468"} err="failed to get container status \"c11dbc9dfcb2815404820823563579d13e56abc148d845235b08ffd73a6d1468\": rpc error: code = NotFound desc = could not find container \"c11dbc9dfcb2815404820823563579d13e56abc148d845235b08ffd73a6d1468\": container with ID starting with c11dbc9dfcb2815404820823563579d13e56abc148d845235b08ffd73a6d1468 not found: ID does not exist"
Jul 01 18:40:49.301140 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:49.301119    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"]
Jul 01 18:40:49.304914 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:49.304888    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1eb4-predictor-5bf4ddd5b9-j8vpj"]
Jul 01 18:40:50.269704 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:50.269669    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:40:50.270229 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:50.270193    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 18:40:50.820851 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:40:50.820812    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" path="/var/lib/kubelet/pods/d3410be8-a262-4434-beab-ca64a7d2774f/volumes"
Jul 01 18:41:00.271234 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:00.271176    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 18:41:10.270430 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:10.270382    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 18:41:18.898827 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:18.898793    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:41:18.905287 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:18.905262    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:41:20.271011 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:20.270951    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 18:41:30.271200 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:30.271163    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:41:37.824777 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:37.824743    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"]
Jul 01 18:41:37.825198 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:37.825016    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kserve-container" containerID="cri-o://ed4cb48ea3c0c7bb1345ca4344b8ab1c51bc5a1df95e7a267336aeedbae941f2" gracePeriod=30
Jul 01 18:41:37.825198 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:37.825056    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kube-rbac-proxy" containerID="cri-o://1c0540b136417949d34c8f3d3dc51413fe681deca84b49258526c1a79c56c17e" gracePeriod=30
Jul 01 18:41:37.989884 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:37.989848    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"]
Jul 01 18:41:37.990273 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:37.990254    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kube-rbac-proxy"
Jul 01 18:41:37.990373 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:37.990276    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kube-rbac-proxy"
Jul 01 18:41:37.990373 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:37.990298    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kserve-container"
Jul 01 18:41:37.990373 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:37.990306    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kserve-container"
Jul 01 18:41:37.990524 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:37.990387    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kserve-container"
Jul 01 18:41:37.990524 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:37.990401    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3410be8-a262-4434-beab-ca64a7d2774f" containerName="kube-rbac-proxy"
Jul 01 18:41:37.993436 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:37.993411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:37.995970 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:37.995944    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-60085-predictor-serving-cert\""
Jul 01 18:41:37.996127 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:37.995985    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-60085-kube-rbac-proxy-sar-config\""
Jul 01 18:41:38.004628 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:38.004603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"]
Jul 01 18:41:38.051682 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:38.051632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a6ee025-2df3-42de-9e35-92c6163c8229-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-60085-predictor-6857fddfd9-jg6vd\" (UID: \"9a6ee025-2df3-42de-9e35-92c6163c8229\") " pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:38.051682 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:38.051685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a6ee025-2df3-42de-9e35-92c6163c8229-proxy-tls\") pod \"error-404-isvc-60085-predictor-6857fddfd9-jg6vd\" (UID: \"9a6ee025-2df3-42de-9e35-92c6163c8229\") " pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:38.051921 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:38.051786    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkjkm\" (UniqueName: \"kubernetes.io/projected/9a6ee025-2df3-42de-9e35-92c6163c8229-kube-api-access-bkjkm\") pod \"error-404-isvc-60085-predictor-6857fddfd9-jg6vd\" (UID: \"9a6ee025-2df3-42de-9e35-92c6163c8229\") " pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:38.152947 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:38.152842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a6ee025-2df3-42de-9e35-92c6163c8229-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-60085-predictor-6857fddfd9-jg6vd\" (UID: \"9a6ee025-2df3-42de-9e35-92c6163c8229\") " pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:38.152947 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:38.152892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a6ee025-2df3-42de-9e35-92c6163c8229-proxy-tls\") pod \"error-404-isvc-60085-predictor-6857fddfd9-jg6vd\" (UID: \"9a6ee025-2df3-42de-9e35-92c6163c8229\") " pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:38.152947 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:38.152952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkjkm\" (UniqueName: \"kubernetes.io/projected/9a6ee025-2df3-42de-9e35-92c6163c8229-kube-api-access-bkjkm\") pod \"error-404-isvc-60085-predictor-6857fddfd9-jg6vd\" (UID: \"9a6ee025-2df3-42de-9e35-92c6163c8229\") " pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:38.153238 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:41:38.153092    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-60085-predictor-serving-cert: secret "error-404-isvc-60085-predictor-serving-cert" not found
Jul 01 18:41:38.153238 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:41:38.153159    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6ee025-2df3-42de-9e35-92c6163c8229-proxy-tls podName:9a6ee025-2df3-42de-9e35-92c6163c8229 nodeName:}" failed. No retries permitted until 2026-07-01 18:41:38.653142061 +0000 UTC m=+2120.220427018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9a6ee025-2df3-42de-9e35-92c6163c8229-proxy-tls") pod "error-404-isvc-60085-predictor-6857fddfd9-jg6vd" (UID: "9a6ee025-2df3-42de-9e35-92c6163c8229") : secret "error-404-isvc-60085-predictor-serving-cert" not found
Jul 01 18:41:38.153570 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:38.153550    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a6ee025-2df3-42de-9e35-92c6163c8229-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-60085-predictor-6857fddfd9-jg6vd\" (UID: \"9a6ee025-2df3-42de-9e35-92c6163c8229\") " pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:38.163692 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:38.163658    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkjkm\" (UniqueName: \"kubernetes.io/projected/9a6ee025-2df3-42de-9e35-92c6163c8229-kube-api-access-bkjkm\") pod \"error-404-isvc-60085-predictor-6857fddfd9-jg6vd\" (UID: \"9a6ee025-2df3-42de-9e35-92c6163c8229\") " pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:38.461872 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:38.461782    2578 generic.go:356] "Generic (PLEG): container finished" podID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerID="1c0540b136417949d34c8f3d3dc51413fe681deca84b49258526c1a79c56c17e" exitCode=2
Jul 01 18:41:38.461872 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:38.461858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" event={"ID":"cac7e284-aea5-4817-b2ea-927dbc78868a","Type":"ContainerDied","Data":"1c0540b136417949d34c8f3d3dc51413fe681deca84b49258526c1a79c56c17e"}
Jul 01 18:41:38.656848 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:38.656805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a6ee025-2df3-42de-9e35-92c6163c8229-proxy-tls\") pod \"error-404-isvc-60085-predictor-6857fddfd9-jg6vd\" (UID: \"9a6ee025-2df3-42de-9e35-92c6163c8229\") " pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:38.659347 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:38.659315    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a6ee025-2df3-42de-9e35-92c6163c8229-proxy-tls\") pod \"error-404-isvc-60085-predictor-6857fddfd9-jg6vd\" (UID: \"9a6ee025-2df3-42de-9e35-92c6163c8229\") " pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:38.904339 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:38.904301    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:39.040256 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:39.040222    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"]
Jul 01 18:41:39.043068 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:41:39.043042    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6ee025_2df3_42de_9e35_92c6163c8229.slice/crio-1e9ae301d49d18519a8b2ed556203b81c7e5ff6c702ea20dc23e212503651fb5 WatchSource:0}: Error finding container 1e9ae301d49d18519a8b2ed556203b81c7e5ff6c702ea20dc23e212503651fb5: Status 404 returned error can't find the container with id 1e9ae301d49d18519a8b2ed556203b81c7e5ff6c702ea20dc23e212503651fb5
Jul 01 18:41:39.468145 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:39.468045    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" event={"ID":"9a6ee025-2df3-42de-9e35-92c6163c8229","Type":"ContainerStarted","Data":"9e63d0962a98e07f2eac320a40d5edd410a2c4867b02d85493ba28dbfa49df32"}
Jul 01 18:41:39.468306 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:39.468146    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:39.468306 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:39.468181    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:39.468306 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:39.468205    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" event={"ID":"9a6ee025-2df3-42de-9e35-92c6163c8229","Type":"ContainerStarted","Data":"4ffc2fe1df606fbf7932a5dcd0c3e93d6eac61ace525d42239503f54c8ed73e7"}
Jul 01 18:41:39.468306 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:39.468233    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" event={"ID":"9a6ee025-2df3-42de-9e35-92c6163c8229","Type":"ContainerStarted","Data":"1e9ae301d49d18519a8b2ed556203b81c7e5ff6c702ea20dc23e212503651fb5"}
Jul 01 18:41:39.469600 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:39.469570    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 18:41:39.488010 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:39.487939    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" podStartSLOduration=2.487924794 podStartE2EDuration="2.487924794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:41:39.486138603 +0000 UTC m=+2121.053423582" watchObservedRunningTime="2026-07-01 18:41:39.487924794 +0000 UTC m=+2121.055209772"
Jul 01 18:41:40.471968 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:40.471927    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 18:41:41.454574 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:41.454471    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 01 18:41:41.459872 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:41.459843    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 18:41:41.477111 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:41.477080    2578 generic.go:356] "Generic (PLEG): container finished" podID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerID="ed4cb48ea3c0c7bb1345ca4344b8ab1c51bc5a1df95e7a267336aeedbae941f2" exitCode=0
Jul 01 18:41:41.477515 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:41.477157    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" event={"ID":"cac7e284-aea5-4817-b2ea-927dbc78868a","Type":"ContainerDied","Data":"ed4cb48ea3c0c7bb1345ca4344b8ab1c51bc5a1df95e7a267336aeedbae941f2"}
Jul 01 18:41:41.782810 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:41.782779    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:41:41.878152 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:41.878110    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p66xq\" (UniqueName: \"kubernetes.io/projected/cac7e284-aea5-4817-b2ea-927dbc78868a-kube-api-access-p66xq\") pod \"cac7e284-aea5-4817-b2ea-927dbc78868a\" (UID: \"cac7e284-aea5-4817-b2ea-927dbc78868a\") "
Jul 01 18:41:41.878152 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:41.878151    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cac7e284-aea5-4817-b2ea-927dbc78868a-kube-rbac-proxy-sar-config\") pod \"cac7e284-aea5-4817-b2ea-927dbc78868a\" (UID: \"cac7e284-aea5-4817-b2ea-927dbc78868a\") "
Jul 01 18:41:41.878347 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:41.878189    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cac7e284-aea5-4817-b2ea-927dbc78868a-proxy-tls\") pod \"cac7e284-aea5-4817-b2ea-927dbc78868a\" (UID: \"cac7e284-aea5-4817-b2ea-927dbc78868a\") "
Jul 01 18:41:41.878547 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:41.878519    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac7e284-aea5-4817-b2ea-927dbc78868a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cac7e284-aea5-4817-b2ea-927dbc78868a" (UID: "cac7e284-aea5-4817-b2ea-927dbc78868a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:41:41.880357 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:41.880324    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac7e284-aea5-4817-b2ea-927dbc78868a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cac7e284-aea5-4817-b2ea-927dbc78868a" (UID: "cac7e284-aea5-4817-b2ea-927dbc78868a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:41:41.880481 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:41.880355    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac7e284-aea5-4817-b2ea-927dbc78868a-kube-api-access-p66xq" (OuterVolumeSpecName: "kube-api-access-p66xq") pod "cac7e284-aea5-4817-b2ea-927dbc78868a" (UID: "cac7e284-aea5-4817-b2ea-927dbc78868a"). InnerVolumeSpecName "kube-api-access-p66xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:41:41.979031 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:41.978975    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p66xq\" (UniqueName: \"kubernetes.io/projected/cac7e284-aea5-4817-b2ea-927dbc78868a-kube-api-access-p66xq\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:41:41.979031 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:41.979021    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cac7e284-aea5-4817-b2ea-927dbc78868a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:41:41.979031 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:41.979035    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cac7e284-aea5-4817-b2ea-927dbc78868a-proxy-tls\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:41:42.483095 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:42.483056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl" event={"ID":"cac7e284-aea5-4817-b2ea-927dbc78868a","Type":"ContainerDied","Data":"f9557b8fceecb34375525ab5127b237343b39baea718e76eb3615ca0336aab23"}
Jul 01 18:41:42.483514 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:42.483133    2578 scope.go:117] "RemoveContainer" containerID="1c0540b136417949d34c8f3d3dc51413fe681deca84b49258526c1a79c56c17e"
Jul 01 18:41:42.483514 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:42.483073    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"
Jul 01 18:41:42.491678 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:42.491659    2578 scope.go:117] "RemoveContainer" containerID="ed4cb48ea3c0c7bb1345ca4344b8ab1c51bc5a1df95e7a267336aeedbae941f2"
Jul 01 18:41:42.505562 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:42.505534    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"]
Jul 01 18:41:42.509876 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:42.509850    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-293b3-predictor-864bd98777-svbdl"]
Jul 01 18:41:42.821582 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:42.821545    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" path="/var/lib/kubelet/pods/cac7e284-aea5-4817-b2ea-927dbc78868a/volumes"
Jul 01 18:41:45.476481 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:45.476453    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:41:45.477103 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:45.477075    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 18:41:52.325396 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.325356    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"]
Jul 01 18:41:52.325957 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.325659    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kube-rbac-proxy"
Jul 01 18:41:52.325957 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.325675    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kube-rbac-proxy"
Jul 01 18:41:52.325957 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.325696    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kserve-container"
Jul 01 18:41:52.325957 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.325702    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kserve-container"
Jul 01 18:41:52.325957 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.325751    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kserve-container"
Jul 01 18:41:52.325957 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.325761    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac7e284-aea5-4817-b2ea-927dbc78868a" containerName="kube-rbac-proxy"
Jul 01 18:41:52.330509 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.330472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:52.332952 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.332919    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1f8d1-predictor-serving-cert\""
Jul 01 18:41:52.333076 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.332987    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1f8d1-kube-rbac-proxy-sar-config\""
Jul 01 18:41:52.338903 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.338871    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"]
Jul 01 18:41:52.362757 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.362723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71a450af-7792-4cdf-a498-22f66f8e38ff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj\" (UID: \"71a450af-7792-4cdf-a498-22f66f8e38ff\") " pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:52.362948 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.362763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x48lg\" (UniqueName: \"kubernetes.io/projected/71a450af-7792-4cdf-a498-22f66f8e38ff-kube-api-access-x48lg\") pod \"error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj\" (UID: \"71a450af-7792-4cdf-a498-22f66f8e38ff\") " pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:52.362948 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.362814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71a450af-7792-4cdf-a498-22f66f8e38ff-proxy-tls\") pod \"error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj\" (UID: \"71a450af-7792-4cdf-a498-22f66f8e38ff\") " pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:52.382013 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.381952    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"]
Jul 01 18:41:52.382283 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.382255    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kserve-container" containerID="cri-o://8e8e654a22da92257dd70e8989b5a34aeee736a30108e70fcb3c11b89b194228" gracePeriod=30
Jul 01 18:41:52.382371 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.382319    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kube-rbac-proxy" containerID="cri-o://6ea81fa88af1fc42dd82332d7abcbc8d91f7089cbb3411ec34da9335decd086a" gracePeriod=30
Jul 01 18:41:52.463575 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.463525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71a450af-7792-4cdf-a498-22f66f8e38ff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj\" (UID: \"71a450af-7792-4cdf-a498-22f66f8e38ff\") " pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:52.463575 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.463577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x48lg\" (UniqueName: \"kubernetes.io/projected/71a450af-7792-4cdf-a498-22f66f8e38ff-kube-api-access-x48lg\") pod \"error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj\" (UID: \"71a450af-7792-4cdf-a498-22f66f8e38ff\") " pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:52.463853 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.463612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71a450af-7792-4cdf-a498-22f66f8e38ff-proxy-tls\") pod \"error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj\" (UID: \"71a450af-7792-4cdf-a498-22f66f8e38ff\") " pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:52.463853 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:41:52.463729    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-serving-cert: secret "error-404-isvc-1f8d1-predictor-serving-cert" not found
Jul 01 18:41:52.463853 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:41:52.463818    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a450af-7792-4cdf-a498-22f66f8e38ff-proxy-tls podName:71a450af-7792-4cdf-a498-22f66f8e38ff nodeName:}" failed. No retries permitted until 2026-07-01 18:41:52.963795141 +0000 UTC m=+2134.531080099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71a450af-7792-4cdf-a498-22f66f8e38ff-proxy-tls") pod "error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" (UID: "71a450af-7792-4cdf-a498-22f66f8e38ff") : secret "error-404-isvc-1f8d1-predictor-serving-cert" not found
Jul 01 18:41:52.464227 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.464206    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71a450af-7792-4cdf-a498-22f66f8e38ff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj\" (UID: \"71a450af-7792-4cdf-a498-22f66f8e38ff\") " pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:52.472092 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.472068    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x48lg\" (UniqueName: \"kubernetes.io/projected/71a450af-7792-4cdf-a498-22f66f8e38ff-kube-api-access-x48lg\") pod \"error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj\" (UID: \"71a450af-7792-4cdf-a498-22f66f8e38ff\") " pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:52.523640 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.523609    2578 generic.go:356] "Generic (PLEG): container finished" podID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerID="6ea81fa88af1fc42dd82332d7abcbc8d91f7089cbb3411ec34da9335decd086a" exitCode=2
Jul 01 18:41:52.523798 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.523681    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" event={"ID":"3f64d70e-353c-4fe5-bdbc-52e32677e226","Type":"ContainerDied","Data":"6ea81fa88af1fc42dd82332d7abcbc8d91f7089cbb3411ec34da9335decd086a"}
Jul 01 18:41:52.967260 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:52.967209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71a450af-7792-4cdf-a498-22f66f8e38ff-proxy-tls\") pod \"error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj\" (UID: \"71a450af-7792-4cdf-a498-22f66f8e38ff\") " pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:52.967447 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:41:52.967366    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-serving-cert: secret "error-404-isvc-1f8d1-predictor-serving-cert" not found
Jul 01 18:41:52.967447 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:41:52.967439    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a450af-7792-4cdf-a498-22f66f8e38ff-proxy-tls podName:71a450af-7792-4cdf-a498-22f66f8e38ff nodeName:}" failed. No retries permitted until 2026-07-01 18:41:53.967424243 +0000 UTC m=+2135.534709201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71a450af-7792-4cdf-a498-22f66f8e38ff-proxy-tls") pod "error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" (UID: "71a450af-7792-4cdf-a498-22f66f8e38ff") : secret "error-404-isvc-1f8d1-predictor-serving-cert" not found
Jul 01 18:41:53.975498 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:53.975454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71a450af-7792-4cdf-a498-22f66f8e38ff-proxy-tls\") pod \"error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj\" (UID: \"71a450af-7792-4cdf-a498-22f66f8e38ff\") " pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:53.977943 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:53.977912    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71a450af-7792-4cdf-a498-22f66f8e38ff-proxy-tls\") pod \"error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj\" (UID: \"71a450af-7792-4cdf-a498-22f66f8e38ff\") " pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:54.143082 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:54.143044    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:54.280785 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:54.280751    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"]
Jul 01 18:41:54.283457 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:41:54.283424    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a450af_7792_4cdf_a498_22f66f8e38ff.slice/crio-e2c1d2001787a906d466114d6d85f10e94d881c233a154ba84f3521c0ff6c51b WatchSource:0}: Error finding container e2c1d2001787a906d466114d6d85f10e94d881c233a154ba84f3521c0ff6c51b: Status 404 returned error can't find the container with id e2c1d2001787a906d466114d6d85f10e94d881c233a154ba84f3521c0ff6c51b
Jul 01 18:41:54.534297 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:54.534257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" event={"ID":"71a450af-7792-4cdf-a498-22f66f8e38ff","Type":"ContainerStarted","Data":"1042cbdf2eb3a3d7332d448f1444439c11592a83bc0cc334d73f8b77002e4efe"}
Jul 01 18:41:54.534509 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:54.534336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" event={"ID":"71a450af-7792-4cdf-a498-22f66f8e38ff","Type":"ContainerStarted","Data":"ad856e6b9e5453a689801fb7c1e51435b4472b22e85fe4552cb915ce9ec6ed1c"}
Jul 01 18:41:54.534509 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:54.534371    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" event={"ID":"71a450af-7792-4cdf-a498-22f66f8e38ff","Type":"ContainerStarted","Data":"e2c1d2001787a906d466114d6d85f10e94d881c233a154ba84f3521c0ff6c51b"}
Jul 01 18:41:54.534509 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:54.534401    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:54.534509 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:54.534431    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:41:54.536143 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:54.536089    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 18:41:54.551764 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:54.551712    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" podStartSLOduration=2.551696031 podStartE2EDuration="2.551696031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:41:54.550223053 +0000 UTC m=+2136.117508034" watchObservedRunningTime="2026-07-01 18:41:54.551696031 +0000 UTC m=+2136.118981009"
Jul 01 18:41:55.265449 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:55.265411    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 01 18:41:55.477830 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:55.477792    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 18:41:55.538214 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:55.538174    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 18:41:55.938048 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:55.938019    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:41:55.991738 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:55.991704    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f64d70e-353c-4fe5-bdbc-52e32677e226-proxy-tls\") pod \"3f64d70e-353c-4fe5-bdbc-52e32677e226\" (UID: \"3f64d70e-353c-4fe5-bdbc-52e32677e226\") "
Jul 01 18:41:55.991919 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:55.991752    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f64d70e-353c-4fe5-bdbc-52e32677e226-kube-rbac-proxy-sar-config\") pod \"3f64d70e-353c-4fe5-bdbc-52e32677e226\" (UID: \"3f64d70e-353c-4fe5-bdbc-52e32677e226\") "
Jul 01 18:41:55.991919 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:55.991785    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5vj7\" (UniqueName: \"kubernetes.io/projected/3f64d70e-353c-4fe5-bdbc-52e32677e226-kube-api-access-k5vj7\") pod \"3f64d70e-353c-4fe5-bdbc-52e32677e226\" (UID: \"3f64d70e-353c-4fe5-bdbc-52e32677e226\") "
Jul 01 18:41:55.992219 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:55.992188    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f64d70e-353c-4fe5-bdbc-52e32677e226-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3f64d70e-353c-4fe5-bdbc-52e32677e226" (UID: "3f64d70e-353c-4fe5-bdbc-52e32677e226"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:41:55.993934 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:55.993899    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f64d70e-353c-4fe5-bdbc-52e32677e226-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3f64d70e-353c-4fe5-bdbc-52e32677e226" (UID: "3f64d70e-353c-4fe5-bdbc-52e32677e226"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:41:55.993934 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:55.993903    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f64d70e-353c-4fe5-bdbc-52e32677e226-kube-api-access-k5vj7" (OuterVolumeSpecName: "kube-api-access-k5vj7") pod "3f64d70e-353c-4fe5-bdbc-52e32677e226" (UID: "3f64d70e-353c-4fe5-bdbc-52e32677e226"). InnerVolumeSpecName "kube-api-access-k5vj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:41:56.092336 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.092234    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f64d70e-353c-4fe5-bdbc-52e32677e226-proxy-tls\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:41:56.092336 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.092271    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f64d70e-353c-4fe5-bdbc-52e32677e226-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:41:56.092336 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.092287    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5vj7\" (UniqueName: \"kubernetes.io/projected/3f64d70e-353c-4fe5-bdbc-52e32677e226-kube-api-access-k5vj7\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:41:56.542834 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.542803    2578 generic.go:356] "Generic (PLEG): container finished" podID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerID="8e8e654a22da92257dd70e8989b5a34aeee736a30108e70fcb3c11b89b194228" exitCode=0
Jul 01 18:41:56.543248 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.542858    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"
Jul 01 18:41:56.543248 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.542900    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" event={"ID":"3f64d70e-353c-4fe5-bdbc-52e32677e226","Type":"ContainerDied","Data":"8e8e654a22da92257dd70e8989b5a34aeee736a30108e70fcb3c11b89b194228"}
Jul 01 18:41:56.543248 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.542966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls" event={"ID":"3f64d70e-353c-4fe5-bdbc-52e32677e226","Type":"ContainerDied","Data":"b5871673604453383eb3256c635e22f38162371f7596d62589192324541aa433"}
Jul 01 18:41:56.543248 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.543011    2578 scope.go:117] "RemoveContainer" containerID="6ea81fa88af1fc42dd82332d7abcbc8d91f7089cbb3411ec34da9335decd086a"
Jul 01 18:41:56.551574 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.551550    2578 scope.go:117] "RemoveContainer" containerID="8e8e654a22da92257dd70e8989b5a34aeee736a30108e70fcb3c11b89b194228"
Jul 01 18:41:56.564653 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.564628    2578 scope.go:117] "RemoveContainer" containerID="6ea81fa88af1fc42dd82332d7abcbc8d91f7089cbb3411ec34da9335decd086a"
Jul 01 18:41:56.564740 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.564718    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"]
Jul 01 18:41:56.565043 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:41:56.565019    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea81fa88af1fc42dd82332d7abcbc8d91f7089cbb3411ec34da9335decd086a\": container with ID starting with 6ea81fa88af1fc42dd82332d7abcbc8d91f7089cbb3411ec34da9335decd086a not found: ID does not exist" containerID="6ea81fa88af1fc42dd82332d7abcbc8d91f7089cbb3411ec34da9335decd086a"
Jul 01 18:41:56.565097 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.565053    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea81fa88af1fc42dd82332d7abcbc8d91f7089cbb3411ec34da9335decd086a"} err="failed to get container status \"6ea81fa88af1fc42dd82332d7abcbc8d91f7089cbb3411ec34da9335decd086a\": rpc error: code = NotFound desc = could not find container \"6ea81fa88af1fc42dd82332d7abcbc8d91f7089cbb3411ec34da9335decd086a\": container with ID starting with 6ea81fa88af1fc42dd82332d7abcbc8d91f7089cbb3411ec34da9335decd086a not found: ID does not exist"
Jul 01 18:41:56.565097 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.565072    2578 scope.go:117] "RemoveContainer" containerID="8e8e654a22da92257dd70e8989b5a34aeee736a30108e70fcb3c11b89b194228"
Jul 01 18:41:56.565365 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:41:56.565348    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8e654a22da92257dd70e8989b5a34aeee736a30108e70fcb3c11b89b194228\": container with ID starting with 8e8e654a22da92257dd70e8989b5a34aeee736a30108e70fcb3c11b89b194228 not found: ID does not exist" containerID="8e8e654a22da92257dd70e8989b5a34aeee736a30108e70fcb3c11b89b194228"
Jul 01 18:41:56.565406 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.565369    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8e654a22da92257dd70e8989b5a34aeee736a30108e70fcb3c11b89b194228"} err="failed to get container status \"8e8e654a22da92257dd70e8989b5a34aeee736a30108e70fcb3c11b89b194228\": rpc error: code = NotFound desc = could not find container \"8e8e654a22da92257dd70e8989b5a34aeee736a30108e70fcb3c11b89b194228\": container with ID starting with 8e8e654a22da92257dd70e8989b5a34aeee736a30108e70fcb3c11b89b194228 not found: ID does not exist"
Jul 01 18:41:56.569026 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.568986    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbcf2-predictor-748c594f6f-tlrls"]
Jul 01 18:41:56.820342 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:41:56.820263    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" path="/var/lib/kubelet/pods/3f64d70e-353c-4fe5-bdbc-52e32677e226/volumes"
Jul 01 18:42:00.542824 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:42:00.542796    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:42:00.543344 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:42:00.543316    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 18:42:05.477723 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:42:05.477684    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 18:42:10.543500 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:42:10.543464    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 18:42:15.477935 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:42:15.477845    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 18:42:20.543482 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:42:20.543441    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 18:42:25.477183 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:42:25.477140    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 18:42:30.543714 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:42:30.543674    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 18:42:35.478146 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:42:35.478116    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:42:40.543721 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:42:40.543669    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 18:42:50.544075 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:42:50.544045    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:46:18.923191 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:46:18.923164    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:46:18.927868 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:46:18.927841    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:51:07.206394 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:07.206359    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"]
Jul 01 18:51:07.206912 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:07.206667    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kserve-container" containerID="cri-o://ad856e6b9e5453a689801fb7c1e51435b4472b22e85fe4552cb915ce9ec6ed1c" gracePeriod=30
Jul 01 18:51:07.206912 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:07.206688    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kube-rbac-proxy" containerID="cri-o://1042cbdf2eb3a3d7332d448f1444439c11592a83bc0cc334d73f8b77002e4efe" gracePeriod=30
Jul 01 18:51:07.556968 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:07.556929    2578 generic.go:356] "Generic (PLEG): container finished" podID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerID="1042cbdf2eb3a3d7332d448f1444439c11592a83bc0cc334d73f8b77002e4efe" exitCode=2
Jul 01 18:51:07.557155 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:07.557019    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" event={"ID":"71a450af-7792-4cdf-a498-22f66f8e38ff","Type":"ContainerDied","Data":"1042cbdf2eb3a3d7332d448f1444439c11592a83bc0cc334d73f8b77002e4efe"}
Jul 01 18:51:10.354015 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.353949    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:51:10.400490 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.400408    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x48lg\" (UniqueName: \"kubernetes.io/projected/71a450af-7792-4cdf-a498-22f66f8e38ff-kube-api-access-x48lg\") pod \"71a450af-7792-4cdf-a498-22f66f8e38ff\" (UID: \"71a450af-7792-4cdf-a498-22f66f8e38ff\") "
Jul 01 18:51:10.400490 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.400447    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71a450af-7792-4cdf-a498-22f66f8e38ff-proxy-tls\") pod \"71a450af-7792-4cdf-a498-22f66f8e38ff\" (UID: \"71a450af-7792-4cdf-a498-22f66f8e38ff\") "
Jul 01 18:51:10.400490 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.400481    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71a450af-7792-4cdf-a498-22f66f8e38ff-kube-rbac-proxy-sar-config\") pod \"71a450af-7792-4cdf-a498-22f66f8e38ff\" (UID: \"71a450af-7792-4cdf-a498-22f66f8e38ff\") "
Jul 01 18:51:10.400899 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.400872    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a450af-7792-4cdf-a498-22f66f8e38ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71a450af-7792-4cdf-a498-22f66f8e38ff" (UID: "71a450af-7792-4cdf-a498-22f66f8e38ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:51:10.402625 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.402598    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a450af-7792-4cdf-a498-22f66f8e38ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71a450af-7792-4cdf-a498-22f66f8e38ff" (UID: "71a450af-7792-4cdf-a498-22f66f8e38ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:51:10.402725 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.402628    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a450af-7792-4cdf-a498-22f66f8e38ff-kube-api-access-x48lg" (OuterVolumeSpecName: "kube-api-access-x48lg") pod "71a450af-7792-4cdf-a498-22f66f8e38ff" (UID: "71a450af-7792-4cdf-a498-22f66f8e38ff"). InnerVolumeSpecName "kube-api-access-x48lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:51:10.501522 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.501479    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x48lg\" (UniqueName: \"kubernetes.io/projected/71a450af-7792-4cdf-a498-22f66f8e38ff-kube-api-access-x48lg\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:51:10.501522 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.501512    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71a450af-7792-4cdf-a498-22f66f8e38ff-proxy-tls\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:51:10.501522 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.501522    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71a450af-7792-4cdf-a498-22f66f8e38ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:51:10.568417 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.568378    2578 generic.go:356] "Generic (PLEG): container finished" podID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerID="ad856e6b9e5453a689801fb7c1e51435b4472b22e85fe4552cb915ce9ec6ed1c" exitCode=0
Jul 01 18:51:10.568571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.568434    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"
Jul 01 18:51:10.568571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.568447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" event={"ID":"71a450af-7792-4cdf-a498-22f66f8e38ff","Type":"ContainerDied","Data":"ad856e6b9e5453a689801fb7c1e51435b4472b22e85fe4552cb915ce9ec6ed1c"}
Jul 01 18:51:10.568571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.568507    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj" event={"ID":"71a450af-7792-4cdf-a498-22f66f8e38ff","Type":"ContainerDied","Data":"e2c1d2001787a906d466114d6d85f10e94d881c233a154ba84f3521c0ff6c51b"}
Jul 01 18:51:10.568571 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.568535    2578 scope.go:117] "RemoveContainer" containerID="1042cbdf2eb3a3d7332d448f1444439c11592a83bc0cc334d73f8b77002e4efe"
Jul 01 18:51:10.579401 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.579375    2578 scope.go:117] "RemoveContainer" containerID="ad856e6b9e5453a689801fb7c1e51435b4472b22e85fe4552cb915ce9ec6ed1c"
Jul 01 18:51:10.586889 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.586871    2578 scope.go:117] "RemoveContainer" containerID="1042cbdf2eb3a3d7332d448f1444439c11592a83bc0cc334d73f8b77002e4efe"
Jul 01 18:51:10.587194 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:51:10.587175    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1042cbdf2eb3a3d7332d448f1444439c11592a83bc0cc334d73f8b77002e4efe\": container with ID starting with 1042cbdf2eb3a3d7332d448f1444439c11592a83bc0cc334d73f8b77002e4efe not found: ID does not exist" containerID="1042cbdf2eb3a3d7332d448f1444439c11592a83bc0cc334d73f8b77002e4efe"
Jul 01 18:51:10.587274 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.587209    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1042cbdf2eb3a3d7332d448f1444439c11592a83bc0cc334d73f8b77002e4efe"} err="failed to get container status \"1042cbdf2eb3a3d7332d448f1444439c11592a83bc0cc334d73f8b77002e4efe\": rpc error: code = NotFound desc = could not find container \"1042cbdf2eb3a3d7332d448f1444439c11592a83bc0cc334d73f8b77002e4efe\": container with ID starting with 1042cbdf2eb3a3d7332d448f1444439c11592a83bc0cc334d73f8b77002e4efe not found: ID does not exist"
Jul 01 18:51:10.587274 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.587235    2578 scope.go:117] "RemoveContainer" containerID="ad856e6b9e5453a689801fb7c1e51435b4472b22e85fe4552cb915ce9ec6ed1c"
Jul 01 18:51:10.587467 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:51:10.587451    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad856e6b9e5453a689801fb7c1e51435b4472b22e85fe4552cb915ce9ec6ed1c\": container with ID starting with ad856e6b9e5453a689801fb7c1e51435b4472b22e85fe4552cb915ce9ec6ed1c not found: ID does not exist" containerID="ad856e6b9e5453a689801fb7c1e51435b4472b22e85fe4552cb915ce9ec6ed1c"
Jul 01 18:51:10.587505 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.587474    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad856e6b9e5453a689801fb7c1e51435b4472b22e85fe4552cb915ce9ec6ed1c"} err="failed to get container status \"ad856e6b9e5453a689801fb7c1e51435b4472b22e85fe4552cb915ce9ec6ed1c\": rpc error: code = NotFound desc = could not find container \"ad856e6b9e5453a689801fb7c1e51435b4472b22e85fe4552cb915ce9ec6ed1c\": container with ID starting with ad856e6b9e5453a689801fb7c1e51435b4472b22e85fe4552cb915ce9ec6ed1c not found: ID does not exist"
Jul 01 18:51:10.593799 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.592181    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"]
Jul 01 18:51:10.597969 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.597945    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f8d1-predictor-9c8b6b447-5pwvj"]
Jul 01 18:51:10.820814 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:10.820783    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" path="/var/lib/kubelet/pods/71a450af-7792-4cdf-a498-22f66f8e38ff/volumes"
Jul 01 18:51:18.944105 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:18.943977    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:51:18.961081 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:51:18.949352    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:56:18.965205 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:56:18.965075    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:56:18.970838 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:56:18.970820    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:58:57.374859 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:58:57.374769    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"]
Jul 01 18:58:57.375551 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:58:57.375040    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kserve-container" containerID="cri-o://4ffc2fe1df606fbf7932a5dcd0c3e93d6eac61ace525d42239503f54c8ed73e7" gracePeriod=30
Jul 01 18:58:57.375551 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:58:57.375092    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kube-rbac-proxy" containerID="cri-o://9e63d0962a98e07f2eac320a40d5edd410a2c4867b02d85493ba28dbfa49df32" gracePeriod=30
Jul 01 18:58:58.238943 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:58:58.238897    2578 generic.go:356] "Generic (PLEG): container finished" podID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerID="9e63d0962a98e07f2eac320a40d5edd410a2c4867b02d85493ba28dbfa49df32" exitCode=2
Jul 01 18:58:58.239157 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:58:58.238953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" event={"ID":"9a6ee025-2df3-42de-9e35-92c6163c8229","Type":"ContainerDied","Data":"9e63d0962a98e07f2eac320a40d5edd410a2c4867b02d85493ba28dbfa49df32"}
Jul 01 18:59:00.472354 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:00.472316    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 01 18:59:00.621961 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:00.621928    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:59:00.775294 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:00.775199    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a6ee025-2df3-42de-9e35-92c6163c8229-proxy-tls\") pod \"9a6ee025-2df3-42de-9e35-92c6163c8229\" (UID: \"9a6ee025-2df3-42de-9e35-92c6163c8229\") "
Jul 01 18:59:00.775294 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:00.775272    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a6ee025-2df3-42de-9e35-92c6163c8229-kube-rbac-proxy-sar-config\") pod \"9a6ee025-2df3-42de-9e35-92c6163c8229\" (UID: \"9a6ee025-2df3-42de-9e35-92c6163c8229\") "
Jul 01 18:59:00.775470 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:00.775307    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkjkm\" (UniqueName: \"kubernetes.io/projected/9a6ee025-2df3-42de-9e35-92c6163c8229-kube-api-access-bkjkm\") pod \"9a6ee025-2df3-42de-9e35-92c6163c8229\" (UID: \"9a6ee025-2df3-42de-9e35-92c6163c8229\") "
Jul 01 18:59:00.775689 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:00.775656    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6ee025-2df3-42de-9e35-92c6163c8229-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9a6ee025-2df3-42de-9e35-92c6163c8229" (UID: "9a6ee025-2df3-42de-9e35-92c6163c8229"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:59:00.777384 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:00.777353    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a6ee025-2df3-42de-9e35-92c6163c8229-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9a6ee025-2df3-42de-9e35-92c6163c8229" (UID: "9a6ee025-2df3-42de-9e35-92c6163c8229"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:59:00.777501 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:00.777460    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6ee025-2df3-42de-9e35-92c6163c8229-kube-api-access-bkjkm" (OuterVolumeSpecName: "kube-api-access-bkjkm") pod "9a6ee025-2df3-42de-9e35-92c6163c8229" (UID: "9a6ee025-2df3-42de-9e35-92c6163c8229"). InnerVolumeSpecName "kube-api-access-bkjkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:59:00.876330 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:00.876291    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a6ee025-2df3-42de-9e35-92c6163c8229-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:59:00.876330 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:00.876323    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkjkm\" (UniqueName: \"kubernetes.io/projected/9a6ee025-2df3-42de-9e35-92c6163c8229-kube-api-access-bkjkm\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:59:00.876330 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:00.876334    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a6ee025-2df3-42de-9e35-92c6163c8229-proxy-tls\") on node \"ip-10-0-132-161.ec2.internal\" DevicePath \"\""
Jul 01 18:59:01.251380 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:01.251337    2578 generic.go:356] "Generic (PLEG): container finished" podID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerID="4ffc2fe1df606fbf7932a5dcd0c3e93d6eac61ace525d42239503f54c8ed73e7" exitCode=0
Jul 01 18:59:01.251553 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:01.251394    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"
Jul 01 18:59:01.251553 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:01.251416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" event={"ID":"9a6ee025-2df3-42de-9e35-92c6163c8229","Type":"ContainerDied","Data":"4ffc2fe1df606fbf7932a5dcd0c3e93d6eac61ace525d42239503f54c8ed73e7"}
Jul 01 18:59:01.251553 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:01.251474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd" event={"ID":"9a6ee025-2df3-42de-9e35-92c6163c8229","Type":"ContainerDied","Data":"1e9ae301d49d18519a8b2ed556203b81c7e5ff6c702ea20dc23e212503651fb5"}
Jul 01 18:59:01.251553 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:01.251500    2578 scope.go:117] "RemoveContainer" containerID="9e63d0962a98e07f2eac320a40d5edd410a2c4867b02d85493ba28dbfa49df32"
Jul 01 18:59:01.259384 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:01.259363    2578 scope.go:117] "RemoveContainer" containerID="4ffc2fe1df606fbf7932a5dcd0c3e93d6eac61ace525d42239503f54c8ed73e7"
Jul 01 18:59:01.266701 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:01.266678    2578 scope.go:117] "RemoveContainer" containerID="9e63d0962a98e07f2eac320a40d5edd410a2c4867b02d85493ba28dbfa49df32"
Jul 01 18:59:01.267553 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:59:01.267522    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e63d0962a98e07f2eac320a40d5edd410a2c4867b02d85493ba28dbfa49df32\": container with ID starting with 9e63d0962a98e07f2eac320a40d5edd410a2c4867b02d85493ba28dbfa49df32 not found: ID does not exist" containerID="9e63d0962a98e07f2eac320a40d5edd410a2c4867b02d85493ba28dbfa49df32"
Jul 01 18:59:01.267668 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:01.267558    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e63d0962a98e07f2eac320a40d5edd410a2c4867b02d85493ba28dbfa49df32"} err="failed to get container status \"9e63d0962a98e07f2eac320a40d5edd410a2c4867b02d85493ba28dbfa49df32\": rpc error: code = NotFound desc = could not find container \"9e63d0962a98e07f2eac320a40d5edd410a2c4867b02d85493ba28dbfa49df32\": container with ID starting with 9e63d0962a98e07f2eac320a40d5edd410a2c4867b02d85493ba28dbfa49df32 not found: ID does not exist"
Jul 01 18:59:01.267668 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:01.267581    2578 scope.go:117] "RemoveContainer" containerID="4ffc2fe1df606fbf7932a5dcd0c3e93d6eac61ace525d42239503f54c8ed73e7"
Jul 01 18:59:01.267920 ip-10-0-132-161 kubenswrapper[2578]: E0701 18:59:01.267896    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ffc2fe1df606fbf7932a5dcd0c3e93d6eac61ace525d42239503f54c8ed73e7\": container with ID starting with 4ffc2fe1df606fbf7932a5dcd0c3e93d6eac61ace525d42239503f54c8ed73e7 not found: ID does not exist" containerID="4ffc2fe1df606fbf7932a5dcd0c3e93d6eac61ace525d42239503f54c8ed73e7"
Jul 01 18:59:01.267965 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:01.267928    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ffc2fe1df606fbf7932a5dcd0c3e93d6eac61ace525d42239503f54c8ed73e7"} err="failed to get container status \"4ffc2fe1df606fbf7932a5dcd0c3e93d6eac61ace525d42239503f54c8ed73e7\": rpc error: code = NotFound desc = could not find container \"4ffc2fe1df606fbf7932a5dcd0c3e93d6eac61ace525d42239503f54c8ed73e7\": container with ID starting with 4ffc2fe1df606fbf7932a5dcd0c3e93d6eac61ace525d42239503f54c8ed73e7 not found: ID does not exist"
Jul 01 18:59:01.268595 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:01.268576    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"]
Jul 01 18:59:01.271053 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:01.271034    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-60085-predictor-6857fddfd9-jg6vd"]
Jul 01 18:59:02.820575 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:02.820540    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" path="/var/lib/kubelet/pods/9a6ee025-2df3-42de-9e35-92c6163c8229/volumes"
Jul 01 18:59:32.605846 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:32.605808    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q89rm_c5f3eafe-b7a2-47e9-954a-028746276935/global-pull-secret-syncer/0.log"
Jul 01 18:59:32.756812 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:32.756774    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x6rq6_f7223ef7-a5f6-4690-be25-9bed08246516/konnectivity-agent/0.log"
Jul 01 18:59:32.784493 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:32.784463    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-161.ec2.internal_0de41b0170b5099b66f5351f7020aac1/haproxy/0.log"
Jul 01 18:59:36.350136 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:36.350105    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5j2nn_a8bfa651-2090-43fc-9bf2-33c9314742b9/node-exporter/0.log"
Jul 01 18:59:36.413826 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:36.413793    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5j2nn_a8bfa651-2090-43fc-9bf2-33c9314742b9/kube-rbac-proxy/0.log"
Jul 01 18:59:36.462821 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:36.462796    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5j2nn_a8bfa651-2090-43fc-9bf2-33c9314742b9/init-textfile/0.log"
Jul 01 18:59:38.581255 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:38.581221    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-jm2m4_4c69cbc1-40fc-4abc-909b-bd59ced4b2a2/networking-console-plugin/0.log"
Jul 01 18:59:39.863347 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863313    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"]
Jul 01 18:59:39.863726 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863563    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kube-rbac-proxy"
Jul 01 18:59:39.863726 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863575    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kube-rbac-proxy"
Jul 01 18:59:39.863726 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863582    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kserve-container"
Jul 01 18:59:39.863726 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863587    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kserve-container"
Jul 01 18:59:39.863726 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863595    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kube-rbac-proxy"
Jul 01 18:59:39.863726 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863601    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kube-rbac-proxy"
Jul 01 18:59:39.863726 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863616    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kserve-container"
Jul 01 18:59:39.863726 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863621    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kserve-container"
Jul 01 18:59:39.863726 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863632    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kube-rbac-proxy"
Jul 01 18:59:39.863726 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863638    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kube-rbac-proxy"
Jul 01 18:59:39.863726 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863670    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kserve-container"
Jul 01 18:59:39.863726 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863678    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kserve-container"
Jul 01 18:59:39.863726 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863723    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kserve-container"
Jul 01 18:59:39.863726 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863732    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f64d70e-353c-4fe5-bdbc-52e32677e226" containerName="kube-rbac-proxy"
Jul 01 18:59:39.864213 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863737    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kserve-container"
Jul 01 18:59:39.864213 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863743    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a6ee025-2df3-42de-9e35-92c6163c8229" containerName="kube-rbac-proxy"
Jul 01 18:59:39.864213 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863749    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kserve-container"
Jul 01 18:59:39.864213 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.863755    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a450af-7792-4cdf-a498-22f66f8e38ff" containerName="kube-rbac-proxy"
Jul 01 18:59:39.866473 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.866455    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:39.869108 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.869084    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lpb44\"/\"openshift-service-ca.crt\""
Jul 01 18:59:39.870183 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.870162    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lpb44\"/\"kube-root-ca.crt\""
Jul 01 18:59:39.870301 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.870170    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lpb44\"/\"default-dockercfg-gczn5\""
Jul 01 18:59:39.875667 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.875647    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"]
Jul 01 18:59:39.948591 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.948558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/309e6c4c-7e38-4768-aa66-8255c4cde79f-lib-modules\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:39.948766 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.948602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/309e6c4c-7e38-4768-aa66-8255c4cde79f-sys\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:39.948766 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.948629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/309e6c4c-7e38-4768-aa66-8255c4cde79f-podres\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:39.948766 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.948661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4hqk\" (UniqueName: \"kubernetes.io/projected/309e6c4c-7e38-4768-aa66-8255c4cde79f-kube-api-access-s4hqk\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:39.948766 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:39.948684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/309e6c4c-7e38-4768-aa66-8255c4cde79f-proc\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:40.049622 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.049583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/309e6c4c-7e38-4768-aa66-8255c4cde79f-sys\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:40.049622 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.049624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/309e6c4c-7e38-4768-aa66-8255c4cde79f-podres\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:40.049883 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.049662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4hqk\" (UniqueName: \"kubernetes.io/projected/309e6c4c-7e38-4768-aa66-8255c4cde79f-kube-api-access-s4hqk\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:40.049883 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.049692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/309e6c4c-7e38-4768-aa66-8255c4cde79f-proc\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:40.049883 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.049712    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/309e6c4c-7e38-4768-aa66-8255c4cde79f-sys\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:40.049883 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.049731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/309e6c4c-7e38-4768-aa66-8255c4cde79f-lib-modules\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:40.049883 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.049774    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/309e6c4c-7e38-4768-aa66-8255c4cde79f-proc\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:40.049883 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.049822    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/309e6c4c-7e38-4768-aa66-8255c4cde79f-podres\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:40.049883 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.049875    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/309e6c4c-7e38-4768-aa66-8255c4cde79f-lib-modules\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:40.058683 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.058661    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4hqk\" (UniqueName: \"kubernetes.io/projected/309e6c4c-7e38-4768-aa66-8255c4cde79f-kube-api-access-s4hqk\") pod \"perf-node-gather-daemonset-s2574\" (UID: \"309e6c4c-7e38-4768-aa66-8255c4cde79f\") " pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:40.176700 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.176586    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:40.308749 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.308716    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"]
Jul 01 18:59:40.312127 ip-10-0-132-161 kubenswrapper[2578]: W0701 18:59:40.312100    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod309e6c4c_7e38_4768_aa66_8255c4cde79f.slice/crio-c1f2b74a1f72e78c489893798ff90ef4453f6293b6aaba53ba339eca8c5ab48e WatchSource:0}: Error finding container c1f2b74a1f72e78c489893798ff90ef4453f6293b6aaba53ba339eca8c5ab48e: Status 404 returned error can't find the container with id c1f2b74a1f72e78c489893798ff90ef4453f6293b6aaba53ba339eca8c5ab48e
Jul 01 18:59:40.393814 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.393775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574" event={"ID":"309e6c4c-7e38-4768-aa66-8255c4cde79f","Type":"ContainerStarted","Data":"b2f6cdb948bc61c7c5df21422434e754d33b7289f5bce3567070e3b7dbce5c61"}
Jul 01 18:59:40.394015 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.393833    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574" event={"ID":"309e6c4c-7e38-4768-aa66-8255c4cde79f","Type":"ContainerStarted","Data":"c1f2b74a1f72e78c489893798ff90ef4453f6293b6aaba53ba339eca8c5ab48e"}
Jul 01 18:59:40.394015 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.393875    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:40.409785 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.409735    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574" podStartSLOduration=1.409717852 podStartE2EDuration="1.409717852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:59:40.409104051 +0000 UTC m=+3201.976389044" watchObservedRunningTime="2026-07-01 18:59:40.409717852 +0000 UTC m=+3201.977002830"
Jul 01 18:59:40.998551 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:40.998517    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b7ls7_61840d9c-093c-4b02-8abe-e1db22856cbf/dns/0.log"
Jul 01 18:59:41.020303 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:41.020273    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b7ls7_61840d9c-093c-4b02-8abe-e1db22856cbf/kube-rbac-proxy/0.log"
Jul 01 18:59:41.191651 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:41.191618    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hmcsv_4ce225ff-82c4-49f3-9dd0-b8df3d1b196d/dns-node-resolver/0.log"
Jul 01 18:59:41.699508 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:41.699473    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bbppr_9d1e4c3c-8594-4a71-86fe-9e77d4f4153f/node-ca/0.log"
Jul 01 18:59:42.878025 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:42.877976    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dhrff_fcdd9ed9-9c31-4499-8ace-63a0ec626420/serve-healthcheck-canary/0.log"
Jul 01 18:59:43.389220 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:43.389189    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zxvp7_c7cc4faf-c31f-4cf1-8461-61b5e3827dba/kube-rbac-proxy/0.log"
Jul 01 18:59:43.411386 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:43.411348    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zxvp7_c7cc4faf-c31f-4cf1-8461-61b5e3827dba/exporter/0.log"
Jul 01 18:59:43.432806 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:43.432772    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zxvp7_c7cc4faf-c31f-4cf1-8461-61b5e3827dba/extractor/0.log"
Jul 01 18:59:45.358854 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:45.358819    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-vgv4w_0c96fe95-60ea-492e-91ac-9589e6316a7a/manager/0.log"
Jul 01 18:59:45.542045 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:45.541982    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-zwmnb_7359dbfd-6c9f-405a-aa42-603da2bbc997/manager/0.log"
Jul 01 18:59:46.406774 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:46.406744    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lpb44/perf-node-gather-daemonset-s2574"
Jul 01 18:59:50.974214 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:50.974185    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dg6np_298d79e6-cc04-47e5-98a5-36e2284c5174/kube-multus-additional-cni-plugins/0.log"
Jul 01 18:59:50.996743 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:50.996715    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dg6np_298d79e6-cc04-47e5-98a5-36e2284c5174/egress-router-binary-copy/0.log"
Jul 01 18:59:51.019675 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:51.019653    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dg6np_298d79e6-cc04-47e5-98a5-36e2284c5174/cni-plugins/0.log"
Jul 01 18:59:51.043248 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:51.043210    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dg6np_298d79e6-cc04-47e5-98a5-36e2284c5174/bond-cni-plugin/0.log"
Jul 01 18:59:51.065529 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:51.065503    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dg6np_298d79e6-cc04-47e5-98a5-36e2284c5174/routeoverride-cni/0.log"
Jul 01 18:59:51.091506 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:51.091468    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dg6np_298d79e6-cc04-47e5-98a5-36e2284c5174/whereabouts-cni-bincopy/0.log"
Jul 01 18:59:51.115780 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:51.115751    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dg6np_298d79e6-cc04-47e5-98a5-36e2284c5174/whereabouts-cni/0.log"
Jul 01 18:59:51.497217 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:51.497179    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rc8bn_65be3311-bd63-439e-9980-ff65266cef62/kube-multus/0.log"
Jul 01 18:59:51.551964 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:51.551927    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2r8hd_8936baa9-7b61-4534-b564-a7bb12bf3535/network-metrics-daemon/0.log"
Jul 01 18:59:51.573128 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:51.573097    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2r8hd_8936baa9-7b61-4534-b564-a7bb12bf3535/kube-rbac-proxy/0.log"
Jul 01 18:59:52.667423 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:52.667390    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-controller/0.log"
Jul 01 18:59:52.685439 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:52.685411    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/1.log"
Jul 01 18:59:52.685439 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:52.685438    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovn-acl-logging/0.log"
Jul 01 18:59:52.713437 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:52.713409    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/kube-rbac-proxy-node/0.log"
Jul 01 18:59:52.743870 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:52.743842    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 18:59:52.763476 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:52.763446    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/northd/0.log"
Jul 01 18:59:52.787589 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:52.787567    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/nbdb/0.log"
Jul 01 18:59:52.812970 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:52.812935    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/sbdb/0.log"
Jul 01 18:59:52.878092 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:52.878055    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldwxn_11a3da3b-b7dc-4aed-b261-597c2fc14fdb/ovnkube-controller/0.log"
Jul 01 18:59:54.165758 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:54.165720    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-t4vsn_9ceed3fb-b91c-43c2-8813-7991a3c27659/network-check-target-container/0.log"
Jul 01 18:59:55.114099 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:55.114065    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zgwh2_583c6fb3-3031-43f8-bce9-aa90d151a40b/iptables-alerter/0.log"
Jul 01 18:59:55.866370 ip-10-0-132-161 kubenswrapper[2578]: I0701 18:59:55.866336    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-67lbw_a7450108-384d-45a2-b88d-ac61a546bab4/tuned/0.log"