Jul 01 19:23:12.324379 ip-10-0-138-252 systemd[1]: Starting Kubernetes Kubelet... Jul 01 19:23:12.809697 ip-10-0-138-252 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 01 19:23:12.809697 ip-10-0-138-252 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 01 19:23:12.809697 ip-10-0-138-252 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 01 19:23:12.809697 ip-10-0-138-252 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 01 19:23:12.809697 ip-10-0-138-252 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 01 19:23:12.809697 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.750784 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 01 19:23:12.809697 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756682 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 01 19:23:12.809697 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756700 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 01 19:23:12.809697 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756706 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 01 19:23:12.809697 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756711 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 01 19:23:12.809697 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756718 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 01 19:23:12.809697 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756722 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 01 19:23:12.809697 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756727 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 01 19:23:12.809697 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756731 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756736 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756740 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756744 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756748 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756752 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756757 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756763 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756768 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756772 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756787 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756792 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756796 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756800 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756804 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756808 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756812 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756816 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756820 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756825 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 01 19:23:12.811528 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756829 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756833 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756837 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756842 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756846 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756850 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756854 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756858 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756862 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756867 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756871 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756875 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756879 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756883 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756887 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756892 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756895 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756899 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756903 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756908 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756912 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 01 19:23:12.812404 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756916 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756920 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756924 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756927 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756932 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756936 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756940 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756944 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756948 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756952 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756956 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756960 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756963 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756969 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756973 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756977 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756983 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756987 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756990 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 01 19:23:12.813525 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756994 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.756998 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757002 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757006 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757010 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757014 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757018 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757022 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757026 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757029 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757037 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757043 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757047 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757054 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757061 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757065 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757069 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757074 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 01 19:23:12.814413 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757078 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757081 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757085 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757090 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757094 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757098 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757102 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757105 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757110 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757115 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757119 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757123 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757128 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.757135 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758248 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758258 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758265 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758270 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758275 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 01 19:23:12.815088 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758280 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758284 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758288 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758292 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758298 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758303 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758306 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758310 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758314 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758319 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758323 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758328 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758332 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758335 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758339 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758343 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758347 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758351 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758355 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758359 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 01 19:23:12.815745 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758363 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758367 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758372 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758376 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758380 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758385 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758390 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758394 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758398 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758403 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758406 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758410 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758414 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758418 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758425 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758430 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758435 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758440 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758444 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 01 19:23:12.816393 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758449 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758454 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758458 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758463 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758467 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758471 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758475 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758480 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758484 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758488 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758492 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758496 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758500 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758504 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758509 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758513 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758517 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758521 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758527 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758532 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 01 19:23:12.816965 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758537 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758541 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758546 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758550 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758554 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758558 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758562 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758566 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758569 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758573 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758577 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758583 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758587 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758591 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758595 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758599 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758604 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758608 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758612 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758617 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 01 19:23:12.822532 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758621 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758624 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758628 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758632 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758636 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758640 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758644 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758648 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758661 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758665 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758669 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758673 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758678 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758682 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.758687 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759274 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759280 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759285 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759289 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759293 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 01 19:23:12.823432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759297 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759301 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759305 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759309 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759313 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759317 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759321 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759325 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759329 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759333 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759338 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759342 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759346 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759350 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759353 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759357 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759362 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759366 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759371 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 01 19:23:12.824084 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759375 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759381 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759385 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759389 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759395 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759400 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759406 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759410 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759414 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759418 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759422 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759426 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759430 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759434 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759438 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759442 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759446 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759451 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759456 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759460 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 01 19:23:12.824902 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759464 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759468 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759472 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759477 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759481 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759485 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759490 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759495 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759501 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759505 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759509 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759513 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759517 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759524 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759530 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759534 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759538 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759543 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759547 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 01 19:23:12.825978 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759552 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759556 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759560 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759563 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759568 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759572 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759575 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759579 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759583 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759588 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759593 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759597 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759601 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759605 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759609 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759613 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759618 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759622 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759626 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759629 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 01 19:23:12.826981 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759633 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759637 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759642 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759645 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759650 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759655 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759661 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759665 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759669 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759673 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759677 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759681 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759685 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759690 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759694 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.759698 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759799 2585 flags.go:64] FLAG: --address="0.0.0.0" Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759809 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759824 2585 flags.go:64] FLAG: --anonymous-auth="true" Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759831 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759836 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 01 19:23:12.828174 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759841 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759849 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759856 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759861 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759866 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759871 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759877 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759881 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759886 2585 flags.go:64] FLAG: --cgroup-root="" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759891 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759896 2585 flags.go:64] FLAG: --client-ca-file="" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759901 2585 flags.go:64] FLAG: --cloud-provider="external" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759906 2585 flags.go:64] FLAG: --cluster-dns="[]" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759913 2585 flags.go:64] FLAG: --cluster-domain="" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759918 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759923 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759927 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759935 2585 flags.go:64] FLAG: --container-log-max-files="5" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759941 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759946 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759951 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759956 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759961 2585 flags.go:64] FLAG: --contention-profiling="false" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759965 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 01 19:23:12.828844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759970 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759976 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759981 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759988 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759993 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.759998 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760002 2585 flags.go:64] FLAG: --enable-load-reader="false" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760007 2585 flags.go:64] FLAG: --enable-server="true" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760012 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760019 2585 flags.go:64] FLAG: --event-burst="100" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760024 2585 flags.go:64] FLAG: --event-qps="50" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760028 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760033 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760038 2585 flags.go:64] FLAG: --eviction-hard="" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760044 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760048 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760053 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760057 2585 flags.go:64] FLAG: --eviction-soft="" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760065 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760070 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760075 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760080 2585 flags.go:64] FLAG: --experimental-mounter-path="" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760084 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760089 2585 flags.go:64] FLAG: --fail-swap-on="true" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760093 2585 flags.go:64] FLAG: --feature-gates="" Jul 01 19:23:12.829794 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760102 2585 flags.go:64] FLAG: --file-check-frequency="20s" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760106 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760111 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760117 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760121 2585 flags.go:64] FLAG: --healthz-port="10248" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760126 2585 flags.go:64] FLAG: --help="false" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760130 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-138-252.ec2.internal" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760135 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760158 2585 flags.go:64] FLAG: --http-check-frequency="20s" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760164 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760170 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760175 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760180 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760184 2585 flags.go:64] FLAG: --image-service-endpoint="" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760188 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760193 2585 flags.go:64] FLAG: --kube-api-burst="100" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760197 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760203 2585 flags.go:64] FLAG: --kube-api-qps="50" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760207 2585 flags.go:64] FLAG: --kube-reserved="" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760212 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760216 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760221 2585 flags.go:64] FLAG: --kubelet-cgroups="" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760225 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760230 2585 flags.go:64] FLAG: --lock-file="" Jul 01 19:23:12.830640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760235 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760241 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760246 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760254 2585 flags.go:64] FLAG: --log-json-split-stream="false" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760259 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760264 2585 flags.go:64] FLAG: --log-text-split-stream="false" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760267 2585 flags.go:64] FLAG: --logging-format="text" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760272 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760279 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760283 2585 flags.go:64] FLAG: --manifest-url="" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760288 2585 flags.go:64] FLAG: --manifest-url-header="" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760295 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760300 2585 flags.go:64] FLAG: --max-open-files="1000000" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760307 2585 flags.go:64] FLAG: --max-pods="110" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760312 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760317 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760321 2585 flags.go:64] FLAG: --memory-manager-policy="None" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760326 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760331 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760336 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760341 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760353 2585 flags.go:64] FLAG: --node-status-max-images="50" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760358 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760362 2585 flags.go:64] FLAG: --oom-score-adj="-999" Jul 01 19:23:12.831426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760367 2585 flags.go:64] FLAG: --pod-cidr="" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760373 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760381 2585 flags.go:64] FLAG: --pod-manifest-path="" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760386 2585 flags.go:64] FLAG: --pod-max-pids="-1" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760391 2585 flags.go:64] FLAG: --pods-per-core="0" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760396 2585 flags.go:64] FLAG: --port="10250" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760401 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760406 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06c2a97a07a7f39f5" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760412 2585 flags.go:64] FLAG: --qos-reserved="" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760416 2585 flags.go:64] FLAG: --read-only-port="10255" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760424 2585 flags.go:64] FLAG: --register-node="true" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760428 2585 flags.go:64] FLAG: --register-with-taints="" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760434 2585 flags.go:64] FLAG: --registry-burst="10" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760438 2585 flags.go:64] FLAG: --registry-qps="5" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760443 2585 flags.go:64] FLAG: --reserved-cpus="" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760448 2585 flags.go:64] FLAG: --reserved-memory="" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760486 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760506 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760510 2585 flags.go:64] FLAG: --rotate-certificates="false" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760515 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760519 2585 flags.go:64] FLAG: --runonce="false" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760522 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760527 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760531 2585 flags.go:64] FLAG: --seccomp-default="false" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760535 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 01 19:23:12.832155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760540 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760546 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760552 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760557 2585 flags.go:64] FLAG: --storage-driver-password="root" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760561 2585 flags.go:64] FLAG: --storage-driver-secure="false" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760564 2585 flags.go:64] FLAG: --storage-driver-table="stats" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760568 2585 flags.go:64] FLAG: --storage-driver-user="root" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760571 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760575 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760578 2585 flags.go:64] FLAG: --system-cgroups="" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760581 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760591 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760594 2585 flags.go:64] FLAG: --tls-cert-file="" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760598 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760605 2585 flags.go:64] FLAG: --tls-min-version="" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760608 2585 flags.go:64] FLAG: --tls-private-key-file="" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760611 2585 flags.go:64] FLAG: --topology-manager-policy="none" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760614 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760618 2585 flags.go:64] FLAG: --topology-manager-scope="container" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760621 2585 flags.go:64] FLAG: --v="2" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760628 2585 flags.go:64] FLAG: --version="false" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760636 2585 flags.go:64] FLAG: --vmodule="" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760643 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.760647 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760801 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 01 19:23:12.833028 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760809 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 01 19:23:12.833676 ip-10-0-138-252 systemd[1]: Started Kubernetes Kubelet. Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760814 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760820 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760824 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760827 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760830 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760834 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760837 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760840 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760843 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760846 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760849 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760852 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760855 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760858 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760860 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760863 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760866 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760869 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760871 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 01 19:23:12.834432 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760874 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760878 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760880 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760883 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760886 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760890 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760895 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760899 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760903 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760908 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760913 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760918 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760922 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760925 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760928 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760932 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760935 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760938 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760941 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 01 19:23:12.837614 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760944 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760947 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760950 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760953 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760956 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760958 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760961 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760964 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760967 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760969 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760972 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760975 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760979 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760983 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760988 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760993 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.760997 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761000 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761003 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761006 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 01 19:23:12.838593 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761009 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761012 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761016 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761019 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761022 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761024 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761028 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761031 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761034 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761036 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761039 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761045 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761048 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761051 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761054 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761057 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761059 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761064 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761069 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761073 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 01 19:23:12.840627 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761078 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761083 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761087 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761090 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761093 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761096 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761099 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761102 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761104 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761107 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761110 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761132 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761136 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761141 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761144 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761147 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761149 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761152 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 01 19:23:12.841533 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.761156 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.761171 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.771038 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.771059 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.771082 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.771088 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771166 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771172 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771176 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771179 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771182 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771186 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771189 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771192 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771195 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 01 19:23:12.842990 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771198 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771201 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771204 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771207 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771210 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771212 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771215 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771218 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771221 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771224 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771227 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771229 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771232 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771235 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771238 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771241 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771244 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771247 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771251 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771253 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 01 19:23:12.843667 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771256 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771259 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771262 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771265 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771268 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771271 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771274 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771277 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771279 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771282 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771287 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771291 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771293 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771297 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771300 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771303 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771306 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771309 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771311 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771314 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 01 19:23:12.845475 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771317 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771319 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771322 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771325 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771328 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771331 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771334 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771337 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771340 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771343 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771346 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771348 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771351 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771354 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771357 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771361 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771364 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771366 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771369 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771372 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 01 19:23:12.846273 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771375 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771377 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771380 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771383 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771385 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771388 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771391 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771394 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771396 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771403 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771408 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771411 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771415 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771418 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771421 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771424 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771427 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771430 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771433 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 01 19:23:12.846955 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771437 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 01 19:23:12.962700 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771441 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 01 19:23:12.962700 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771444 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 01 19:23:12.962700 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771447 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 01 19:23:12.962700 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771450 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 01 19:23:12.962700 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771452 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 01 19:23:12.962700 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771455 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 01 19:23:12.962700 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771458 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 01 19:23:12.962700 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771462 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 01 19:23:12.962700 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771465 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 01 19:23:12.962700 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771468 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 01 19:23:12.962700 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.771473 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 01 19:23:12.962700 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771615 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 01 19:23:12.962700 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771622 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 01 19:23:12.962700 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771626 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771630 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771633 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771636 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771638 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771641 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771644 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771646 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771652 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771654 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771657 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771660 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771663 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771666 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771669 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771671 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771674 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771677 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771679 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771682 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 01 19:23:12.984396 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771685 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771688 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771690 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771693 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771696 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771699 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771702 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771707 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771710 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771714 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771716 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771719 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771722 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771725 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771728 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771731 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771734 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771737 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771739 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771742 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 01 19:23:12.986313 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771746 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771749 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771752 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771755 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771757 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771760 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771764 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771768 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771771 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771774 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771777 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771780 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771782 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771785 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771788 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771791 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771793 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771796 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771799 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 01 19:23:12.990926 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771802 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771805 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771808 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771811 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771814 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771816 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771819 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771822 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771825 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771828 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771830 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771833 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771836 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771840 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771843 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771846 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771848 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771852 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771856 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771859 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 01 19:23:12.992416 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771862 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771865 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771867 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771870 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771873 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771875 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771878 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771881 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771883 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771886 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771889 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771892 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771896 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771898 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771901 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771904 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771907 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 01 19:23:12.993763 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:12.771909 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 01 19:23:12.994609 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.771914 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 01 19:23:12.994609 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.772039 2585 server.go:962] "Client rotation is on, will bootstrap in background" Jul 01 19:23:12.994609 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.774933 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 01 19:23:12.994609 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.776665 2585 server.go:1019] "Starting client certificate rotation" Jul 01 19:23:12.994609 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.776774 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 01 19:23:12.994609 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.776816 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 01 19:23:12.994609 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.779367 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 01 19:23:12.994609 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.784584 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 01 19:23:12.994609 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.795798 2585 log.go:25] "Validated CRI v1 runtime API" Jul 01 19:23:12.994609 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.799463 2585 log.go:25] "Validated CRI v1 image API" Jul 01 19:23:12.994609 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.801372 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 01 19:23:12.994609 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.804841 2585 fs.go:135] Filesystem UUIDs: map[3d74d5f9-981d-4dde-9d74-6c3cd46e3d53:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 902ea338-f8ad-4ef0-8dae-a749f33103ac:/dev/nvme0n1p3] Jul 01 19:23:12.996068 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.804862 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 01 19:23:12.996217 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.813491 2585 manager.go:216] Machine: {Timestamp:2026-07-01 19:23:12.812068698 +0000 UTC m=+0.358445329 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100196 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28274607a56245df406b8f01dc8c94 SystemUUID:ec282746-07a5-6245-df40-6b8f01dc8c94 BootID:325f3f8d-04ec-497d-9b4d-6bd1eca55bd7 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6e:8b:05:18:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6e:8b:05:18:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:98:e1:4f:dd:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 01 19:23:12.996217 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.813606 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 01 19:23:12.996217 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.813751 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 01 19:23:12.996217 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.817092 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 01 19:23:12.996217 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.817127 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-252.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 01 19:23:12.996217 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.818432 2585 topology_manager.go:138] "Creating topology manager with none policy" Jul 01 19:23:12.996217 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.818443 2585 container_manager_linux.go:309] "Creating device plugin manager" Jul 01 19:23:12.996217 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.818454 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.819205 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.820096 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.823166 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.823295 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.824061 2585 kubelet.go:486] "Attempting to sync node with API server" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.824076 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.824093 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.824103 2585 kubelet.go:398] "Adding apiserver pod source" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.824114 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.824517 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.824534 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.826853 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.827378 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.828319 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.828350 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.828956 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.828970 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.828977 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.828984 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.828995 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.829003 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.829111 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.829117 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.829124 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.829135 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.829164 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.829174 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.829199 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.829208 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.832756 2585 server.go:1268] "Started kubelet" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.832853 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.832934 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.832964 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.834383 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.835899 2585 server.go:310] "Adding debug handlers to kubelet server" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:12.836855 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.836882 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-252.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 01 19:23:12.996641 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:12.836977 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:12.839188 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-252.ec2.internal.18be40d693d5a64e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-252.ec2.internal,UID:ip-10-0-138-252.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-252.ec2.internal,},FirstTimestamp:2026-07-01 19:23:12.832726606 +0000 UTC m=+0.379103232,LastTimestamp:2026-07-01 19:23:12.832726606 +0000 UTC m=+0.379103232,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-252.ec2.internal,}" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:12.847920 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.848879 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.848892 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:12.848960 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.849541 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.849545 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.849565 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.849693 2585 reconstruct.go:97] "Volume reconstruction finished" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.849704 2585 reconciler.go:29] "Reconciler: start to sync state" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.851329 2585 factory.go:55] Registering systemd factory Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.851347 2585 factory.go:223] Registration of the systemd container factory successfully Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:12.851518 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.851606 2585 factory.go:153] Registering CRI-O factory Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.851619 2585 factory.go:223] Registration of the crio container factory successfully Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:12.851599 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.851672 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.851706 2585 factory.go:103] Registering Raw factory Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.851726 2585 manager.go:1196] Started watching for new ooms in manager Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.852153 2585 manager.go:349] Starting recovery of all containers Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.867452 2585 manager.go:354] Recovery completed Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.871921 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.874005 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.874042 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.874054 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.874663 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.874676 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.874696 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.874668 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.876064 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.876093 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.876114 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID" Jul 01 19:23:12.997856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.877626 2585 policy_none.go:49] "None policy: Start" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.877641 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.877653 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.877925 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v6cdx" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.878817 2585 policy_none.go:47] "Start" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.884357 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v6cdx" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:12.885249 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-252.ec2.internal.18be40d6964bcd20 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-252.ec2.internal,UID:ip-10-0-138-252.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-252.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-252.ec2.internal,},FirstTimestamp:2026-07-01 19:23:12.874024224 +0000 UTC m=+0.420400853,LastTimestamp:2026-07-01 19:23:12.874024224 +0000 UTC m=+0.420400853,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-252.ec2.internal,}" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:12.887202 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-252.ec2.internal.18be40d6964c2930 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-252.ec2.internal,UID:ip-10-0-138-252.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-252.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-252.ec2.internal,},FirstTimestamp:2026-07-01 19:23:12.874047792 +0000 UTC m=+0.420424420,LastTimestamp:2026-07-01 19:23:12.874047792 +0000 UTC m=+0.420424420,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-252.ec2.internal,}" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.930970 2585 manager.go:337] "Starting Device Plugin manager" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:12.931216 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.931229 2585 server.go:87] "Starting device plugin registration server" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.931466 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.931480 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.931619 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.931712 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.931722 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:12.932195 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:12.932252 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-252.ec2.internal\" not found" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.954088 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.955333 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.955368 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.955388 2585 kubelet.go:2448] "Starting kubelet main sync loop" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:12.955426 2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 01 19:23:12.998965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:12.958304 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.034249 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.037295 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.037333 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.037351 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.037377 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.048501 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:13.048522 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-252.ec2.internal\": node \"ip-10-0-138-252.ec2.internal\" not found" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.056548 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal"] Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.056623 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.062319 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.062352 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.062368 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.063801 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.063939 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.063973 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.064569 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.064592 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.064611 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.064623 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.064594 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.064677 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.065917 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.065940 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.066821 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.066848 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.066863 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:13.079017 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:13.102049 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-252.ec2.internal\" not found" node="ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.144526 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:13.106818 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-252.ec2.internal\" not found" node="ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.305555 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.151758 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0ad6b0a84840892a426fb300e0c2d8d6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-252.ec2.internal\" (UID: \"0ad6b0a84840892a426fb300e0c2d8d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.305555 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.151783 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1a212c3b581e59189101d5b1344ee82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"c1a212c3b581e59189101d5b1344ee82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.305555 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.151803 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1a212c3b581e59189101d5b1344ee82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"c1a212c3b581e59189101d5b1344ee82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.305555 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:13.179663 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found" Jul 01 19:23:13.305555 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.252435 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1a212c3b581e59189101d5b1344ee82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"c1a212c3b581e59189101d5b1344ee82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.305555 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.252467 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0ad6b0a84840892a426fb300e0c2d8d6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-252.ec2.internal\" (UID: \"0ad6b0a84840892a426fb300e0c2d8d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.305555 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.252485 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1a212c3b581e59189101d5b1344ee82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"c1a212c3b581e59189101d5b1344ee82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.305555 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.252541 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0ad6b0a84840892a426fb300e0c2d8d6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-252.ec2.internal\" (UID: \"0ad6b0a84840892a426fb300e0c2d8d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.305555 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.252549 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1a212c3b581e59189101d5b1344ee82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"c1a212c3b581e59189101d5b1344ee82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.305555 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.252597 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1a212c3b581e59189101d5b1344ee82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"c1a212c3b581e59189101d5b1344ee82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.305555 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:13.280666 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found" Jul 01 19:23:13.435246 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:13.381578 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found" Jul 01 19:23:13.435246 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.406601 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.435246 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.416213 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" Jul 01 19:23:13.617593 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:13.617556 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found" Jul 01 19:23:13.708437 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.708337 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 01 19:23:13.717681 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:13.717649 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found" Jul 01 19:23:13.777142 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.777111 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 01 19:23:13.777284 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.777261 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 01 19:23:13.777344 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.777304 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 01 19:23:13.818715 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:13.818673 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found" Jul 01 19:23:13.889449 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.889399 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 19:18:13 +0000 UTC" deadline="2028-01-20 19:43:24.30759013 +0000 UTC" Jul 01 19:23:13.889449 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:13.889442 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13632h20m10.418155976s" Jul 01 19:23:13.918995 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:13.918953 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found" Jul 01 19:23:14.020082 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:14.019985 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found" Jul 01 19:23:14.038238 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.038212 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 19:23:14.039735 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.039715 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory" Jul 01 19:23:14.039848 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.039749 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 19:23:14.039848 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.039759 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID" Jul 01 19:23:14.053809 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:14.053779 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad6b0a84840892a426fb300e0c2d8d6.slice/crio-520de46c7c02d0bac923b8d5cbad8b3ee17b3c56df28b90bf51123c72bdf4ab6 WatchSource:0}: Error finding container 520de46c7c02d0bac923b8d5cbad8b3ee17b3c56df28b90bf51123c72bdf4ab6: Status 404 returned error can't find the container with id 520de46c7c02d0bac923b8d5cbad8b3ee17b3c56df28b90bf51123c72bdf4ab6 Jul 01 19:23:14.060651 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.060633 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 01 19:23:14.114908 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:14.114882 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a212c3b581e59189101d5b1344ee82.slice/crio-1b0b26c886f7e2db7fee980ab62db665dc5cf285dfa5b6907a1cf0280880c157 WatchSource:0}: Error finding container 1b0b26c886f7e2db7fee980ab62db665dc5cf285dfa5b6907a1cf0280880c157: Status 404 returned error can't find the container with id 1b0b26c886f7e2db7fee980ab62db665dc5cf285dfa5b6907a1cf0280880c157 Jul 01 19:23:14.119313 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.119294 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 01 19:23:14.149551 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.149514 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" Jul 01 19:23:14.160603 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.160575 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 01 19:23:14.162805 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.162788 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" Jul 01 19:23:14.175523 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.175499 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 01 19:23:14.229733 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.229707 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 01 19:23:14.826313 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.826274 2585 apiserver.go:52] "Watching apiserver" Jul 01 19:23:14.834729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.834702 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 01 19:23:14.835279 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.835250 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-cgn6q","openshift-multus/multus-additional-cni-plugins-plcl5","openshift-multus/network-metrics-daemon-m2flh","openshift-network-diagnostics/network-check-target-csgsb","kube-system/konnectivity-agent-qvn4g","openshift-image-registry/node-ca-926nd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal","openshift-multus/multus-zjd48","openshift-network-operator/iptables-alerter-d6rls","openshift-ovn-kubernetes/ovnkube-node-frt9f","kube-system/global-pull-secret-syncer-qsrzd","kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k","openshift-cluster-node-tuning-operator/tuned-b9f2f"] Jul 01 19:23:14.837310 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.837285 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.838907 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.838883 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:14.839928 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.839826 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 01 19:23:14.840304 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.840259 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 01 19:23:14.840537 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.840521 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 01 19:23:14.840602 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.840541 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 01 19:23:14.840778 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.840756 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dxmdb\"" Jul 01 19:23:14.840848 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.840823 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 01 19:23:14.840933 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.840914 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:14.840933 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.840761 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 01 19:23:14.841030 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:14.840999 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:14.841206 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.841187 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 01 19:23:14.841686 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.841668 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p7grv\"" Jul 01 19:23:14.841838 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.841812 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 01 19:23:14.842092 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.842058 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 01 19:23:14.843098 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.843063 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:14.843214 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:14.843156 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:14.844890 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.844865 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qvn4g" Jul 01 19:23:14.846408 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.846384 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-926nd" Jul 01 19:23:14.847393 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.847372 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 01 19:23:14.847501 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.847374 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 01 19:23:14.847501 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.847467 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z8tsm\"" Jul 01 19:23:14.848311 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.847900 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d6rls" Jul 01 19:23:14.848745 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.848723 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 01 19:23:14.848841 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.848759 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 01 19:23:14.849014 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.848997 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 01 19:23:14.849432 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.849409 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zkvcp\"" Jul 01 19:23:14.849682 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.849663 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 01 19:23:14.850583 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.850563 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.850685 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.850651 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nnkng\"" Jul 01 19:23:14.850743 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.850699 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 01 19:23:14.850743 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.850722 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 01 19:23:14.851128 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.851109 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 01 19:23:14.851741 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.851721 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:14.853172 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.852905 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 01 19:23:14.854495 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.853591 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-krxcm\"" Jul 01 19:23:14.854495 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.853631 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cgn6q" Jul 01 19:23:14.854495 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.854370 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 01 19:23:14.855122 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.855100 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 01 19:23:14.857071 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.857048 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 01 19:23:14.857409 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.857390 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w2jtg\"" Jul 01 19:23:14.857664 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.857646 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 01 19:23:14.857858 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.857837 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 01 19:23:14.857969 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.857950 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cxvcs\"" Jul 01 19:23:14.858108 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.858069 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 01 19:23:14.858358 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.858317 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:14.858471 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.858323 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 01 19:23:14.858471 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:14.858403 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:14.858867 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.858848 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 01 19:23:14.860446 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.860422 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zjd48" Jul 01 19:23:14.862186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.862165 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 01 19:23:14.862878 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.862861 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-flb8r\"" Jul 01 19:23:14.862994 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.862863 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 01 19:23:14.891630 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.891601 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gv2jr" Jul 01 19:23:14.901342 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.901115 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gv2jr" Jul 01 19:23:14.935373 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935343 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ce6745e-2896-41de-b042-33adaac95142-etc-tuned\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.935505 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935401 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ce6745e-2896-41de-b042-33adaac95142-tmp\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.935505 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935428 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dsmf\" (UniqueName: \"kubernetes.io/projected/7ce6745e-2896-41de-b042-33adaac95142-kube-api-access-4dsmf\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.935505 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935449 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-kubelet\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.935505 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935475 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-run-systemd\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.935689 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935518 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-run-openvswitch\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.935689 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935554 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-ovnkube-config\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.935689 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935579 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-kubernetes\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.935689 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935618 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-sysctl-d\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.935689 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935641 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-var-lib-kubelet\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.935689 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935675 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stprv\" (UniqueName: \"kubernetes.io/projected/688cb495-bdbd-40fc-a726-7bc07d269276-kube-api-access-stprv\") pod \"network-metrics-daemon-m2flh\" (UID: \"688cb495-bdbd-40fc-a726-7bc07d269276\") " pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:14.935990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935699 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-systemd-units\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.935990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935715 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-node-log\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.935990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935736 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-run-ovn-kubernetes\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.935990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935762 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtk9j\" (UniqueName: \"kubernetes.io/projected/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-kube-api-access-vtk9j\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.935990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935782 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-host\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.935990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935800 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-device-dir\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:14.935990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935817 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlmld\" (UniqueName: \"kubernetes.io/projected/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-kube-api-access-jlmld\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:14.935990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935840 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w7b5\" (UniqueName: \"kubernetes.io/projected/04296533-7323-4246-bcf2-fc3cd1e387ef-kube-api-access-4w7b5\") pod \"node-ca-926nd\" (UID: \"04296533-7323-4246-bcf2-fc3cd1e387ef\") " pod="openshift-image-registry/node-ca-926nd" Jul 01 19:23:14.935990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935874 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-modprobe-d\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.935990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935906 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs\") pod \"network-metrics-daemon-m2flh\" (UID: \"688cb495-bdbd-40fc-a726-7bc07d269276\") " pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:14.935990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935932 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzpbv\" (UniqueName: \"kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv\") pod \"network-check-target-csgsb\" (UID: \"54990c60-95a2-4a2c-9d4d-690c9994707b\") " pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:14.935990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.935967 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e4f997c-ae5b-4400-874a-89c94b3a5796-konnectivity-ca\") pod \"konnectivity-agent-qvn4g\" (UID: \"2e4f997c-ae5b-4400-874a-89c94b3a5796\") " pod="kube-system/konnectivity-agent-qvn4g" Jul 01 19:23:14.936515 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936004 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fba7d267-ad49-4729-8e5f-737cf617fcd6-iptables-alerter-script\") pod \"iptables-alerter-d6rls\" (UID: \"fba7d267-ad49-4729-8e5f-737cf617fcd6\") " pod="openshift-network-operator/iptables-alerter-d6rls" Jul 01 19:23:14.936515 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936081 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fba7d267-ad49-4729-8e5f-737cf617fcd6-host-slash\") pod \"iptables-alerter-d6rls\" (UID: \"fba7d267-ad49-4729-8e5f-737cf617fcd6\") " pod="openshift-network-operator/iptables-alerter-d6rls" Jul 01 19:23:14.936515 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936109 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04296533-7323-4246-bcf2-fc3cd1e387ef-serviceca\") pod \"node-ca-926nd\" (UID: \"04296533-7323-4246-bcf2-fc3cd1e387ef\") " pod="openshift-image-registry/node-ca-926nd" Jul 01 19:23:14.936515 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936133 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-systemd\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.936515 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936157 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:14.936515 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936182 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e4f997c-ae5b-4400-874a-89c94b3a5796-agent-certs\") pod \"konnectivity-agent-qvn4g\" (UID: \"2e4f997c-ae5b-4400-874a-89c94b3a5796\") " pod="kube-system/konnectivity-agent-qvn4g" Jul 01 19:23:14.936515 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936232 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdfch\" (UniqueName: \"kubernetes.io/projected/fba7d267-ad49-4729-8e5f-737cf617fcd6-kube-api-access-cdfch\") pod \"iptables-alerter-d6rls\" (UID: \"fba7d267-ad49-4729-8e5f-737cf617fcd6\") " pod="openshift-network-operator/iptables-alerter-d6rls" Jul 01 19:23:14.936515 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936265 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-run-ovn\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.936515 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936298 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-ovnkube-script-lib\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.936515 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936322 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-sys\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.936515 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936346 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-cni-netd\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.936515 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936370 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-ovn-node-metrics-cert\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.936515 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936394 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-sysconfig\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.936515 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936423 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-sysctl-conf\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.937207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936462 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-lib-modules\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.937207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936500 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-registration-dir\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:14.937207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936523 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-slash\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.937207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936550 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-run-netns\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.937207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936576 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-var-lib-openvswitch\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.937207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936603 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-cni-bin\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.937207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936625 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-run\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:14.937207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936690 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-socket-dir\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:14.937207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936727 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:14.937207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936772 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-sys-fs\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:14.937207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936805 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-etc-openvswitch\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.937207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936834 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-log-socket\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.937207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936872 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.937850 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936904 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-env-overrides\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:14.937850 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.936928 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04296533-7323-4246-bcf2-fc3cd1e387ef-host\") pod \"node-ca-926nd\" (UID: \"04296533-7323-4246-bcf2-fc3cd1e387ef\") " pod="openshift-image-registry/node-ca-926nd" Jul 01 19:23:14.950944 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.950916 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 01 19:23:14.961608 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.961556 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" event={"ID":"c1a212c3b581e59189101d5b1344ee82","Type":"ContainerStarted","Data":"1b0b26c886f7e2db7fee980ab62db665dc5cf285dfa5b6907a1cf0280880c157"} Jul 01 19:23:14.962875 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:14.962820 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" event={"ID":"0ad6b0a84840892a426fb300e0c2d8d6","Type":"ContainerStarted","Data":"520de46c7c02d0bac923b8d5cbad8b3ee17b3c56df28b90bf51123c72bdf4ab6"} Jul 01 19:23:15.037769 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.037736 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-run-ovn-kubernetes\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.037959 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.037778 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtk9j\" (UniqueName: \"kubernetes.io/projected/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-kube-api-access-vtk9j\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.037959 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.037803 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-host\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.037959 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.037829 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpbv\" (UniqueName: \"kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv\") pod \"network-check-target-csgsb\" (UID: \"54990c60-95a2-4a2c-9d4d-690c9994707b\") " pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:15.037959 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.037854 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fba7d267-ad49-4729-8e5f-737cf617fcd6-iptables-alerter-script\") pod \"iptables-alerter-d6rls\" (UID: \"fba7d267-ad49-4729-8e5f-737cf617fcd6\") " pod="openshift-network-operator/iptables-alerter-d6rls" Jul 01 19:23:15.037959 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.037889 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/90fc2dc7-c463-49e5-b2f5-050c9238f37c-os-release\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.037959 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.037915 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e658cc5b-e0b2-4701-8568-5008df00ed5b-cni-binary-copy\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.037959 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.037942 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w7b5\" (UniqueName: \"kubernetes.io/projected/04296533-7323-4246-bcf2-fc3cd1e387ef-kube-api-access-4w7b5\") pod \"node-ca-926nd\" (UID: \"04296533-7323-4246-bcf2-fc3cd1e387ef\") " pod="openshift-image-registry/node-ca-926nd" Jul 01 19:23:15.038455 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.037985 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-modprobe-d\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.038455 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038002 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs\") pod \"network-metrics-daemon-m2flh\" (UID: \"688cb495-bdbd-40fc-a726-7bc07d269276\") " pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:15.038455 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038025 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdfch\" (UniqueName: \"kubernetes.io/projected/fba7d267-ad49-4729-8e5f-737cf617fcd6-kube-api-access-cdfch\") pod \"iptables-alerter-d6rls\" (UID: \"fba7d267-ad49-4729-8e5f-737cf617fcd6\") " pod="openshift-network-operator/iptables-alerter-d6rls" Jul 01 19:23:15.038455 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038052 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/90fc2dc7-c463-49e5-b2f5-050c9238f37c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.038455 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038075 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a40a5624-aef1-4d71-a765-ccc60715a3d8-hosts-file\") pod \"node-resolver-cgn6q\" (UID: \"a40a5624-aef1-4d71-a765-ccc60715a3d8\") " pod="openshift-dns/node-resolver-cgn6q" Jul 01 19:23:15.038455 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038115 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-multus-conf-dir\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.038455 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038141 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-systemd\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.038455 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038159 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-var-lib-cni-bin\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.038455 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038175 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-var-lib-kubelet\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.038455 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038210 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e658cc5b-e0b2-4701-8568-5008df00ed5b-multus-daemon-config\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.038455 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038237 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-run-ovn\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.038455 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038262 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-kubelet-config\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:15.038455 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038286 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-dbus\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:15.038455 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038309 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:15.039186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038329 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-hostroot\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.039186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038348 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-sysconfig\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.039186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038374 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-lib-modules\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.039186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038396 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlmld\" (UniqueName: \"kubernetes.io/projected/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-kube-api-access-jlmld\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.039186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038412 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-slash\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.039186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038429 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-var-lib-openvswitch\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.039186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038464 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.039186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038489 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90fc2dc7-c463-49e5-b2f5-050c9238f37c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.039186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038515 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/90fc2dc7-c463-49e5-b2f5-050c9238f37c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.039186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038545 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjkph\" (UniqueName: \"kubernetes.io/projected/90fc2dc7-c463-49e5-b2f5-050c9238f37c-kube-api-access-vjkph\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.039186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038562 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40a5624-aef1-4d71-a765-ccc60715a3d8-tmp-dir\") pod \"node-resolver-cgn6q\" (UID: \"a40a5624-aef1-4d71-a765-ccc60715a3d8\") " pod="openshift-dns/node-resolver-cgn6q" Jul 01 19:23:15.039186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038600 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fba7d267-ad49-4729-8e5f-737cf617fcd6-host-slash\") pod \"iptables-alerter-d6rls\" (UID: \"fba7d267-ad49-4729-8e5f-737cf617fcd6\") " pod="openshift-network-operator/iptables-alerter-d6rls" Jul 01 19:23:15.039186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038626 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-etc-openvswitch\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.039186 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038653 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-log-socket\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.039729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038695 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.039729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038721 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-env-overrides\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.039729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038749 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ce6745e-2896-41de-b042-33adaac95142-etc-tuned\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.039729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038772 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ce6745e-2896-41de-b042-33adaac95142-tmp\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.039729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038794 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dsmf\" (UniqueName: \"kubernetes.io/projected/7ce6745e-2896-41de-b042-33adaac95142-kube-api-access-4dsmf\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.039729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038821 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e4f997c-ae5b-4400-874a-89c94b3a5796-agent-certs\") pod \"konnectivity-agent-qvn4g\" (UID: \"2e4f997c-ae5b-4400-874a-89c94b3a5796\") " pod="kube-system/konnectivity-agent-qvn4g" Jul 01 19:23:15.039729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038851 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90fc2dc7-c463-49e5-b2f5-050c9238f37c-system-cni-dir\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.039729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038880 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-run-systemd\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.039729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038907 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-run-openvswitch\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.039729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038933 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-ovnkube-config\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.039729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.038958 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-sysctl-d\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.039729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039000 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-var-lib-kubelet\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.039729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039030 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxxf\" (UniqueName: \"kubernetes.io/projected/a40a5624-aef1-4d71-a765-ccc60715a3d8-kube-api-access-zvxxf\") pod \"node-resolver-cgn6q\" (UID: \"a40a5624-aef1-4d71-a765-ccc60715a3d8\") " pod="openshift-dns/node-resolver-cgn6q" Jul 01 19:23:15.039729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039051 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-systemd-units\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.040372 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039069 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-device-dir\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.040372 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039086 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-os-release\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.040372 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039106 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-run-netns\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.040372 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039122 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-etc-kubernetes\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.040372 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039140 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e4f997c-ae5b-4400-874a-89c94b3a5796-konnectivity-ca\") pod \"konnectivity-agent-qvn4g\" (UID: \"2e4f997c-ae5b-4400-874a-89c94b3a5796\") " pod="kube-system/konnectivity-agent-qvn4g" Jul 01 19:23:15.040372 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039173 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-multus-cni-dir\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.040372 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039191 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-multus-socket-dir-parent\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.040372 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039207 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnmbx\" (UniqueName: \"kubernetes.io/projected/e658cc5b-e0b2-4701-8568-5008df00ed5b-kube-api-access-wnmbx\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.040372 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039487 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04296533-7323-4246-bcf2-fc3cd1e387ef-serviceca\") pod \"node-ca-926nd\" (UID: \"04296533-7323-4246-bcf2-fc3cd1e387ef\") " pod="openshift-image-registry/node-ca-926nd" Jul 01 19:23:15.040372 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039550 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.040372 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039590 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90fc2dc7-c463-49e5-b2f5-050c9238f37c-cnibin\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.040372 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039621 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-run-multus-certs\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.040372 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039871 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-ovnkube-script-lib\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.040372 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039914 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-sys\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.041013 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.040024 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-var-lib-kubelet\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.041013 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.040204 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-sys\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.041013 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.040308 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-host\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.041013 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.040387 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-log-socket\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.041013 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.040446 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.041769 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.040306 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fba7d267-ad49-4729-8e5f-737cf617fcd6-host-slash\") pod \"iptables-alerter-d6rls\" (UID: \"fba7d267-ad49-4729-8e5f-737cf617fcd6\") " pod="openshift-network-operator/iptables-alerter-d6rls" Jul 01 19:23:15.042166 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.042101 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-systemd-units\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.042281 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.042186 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-device-dir\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.042412 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.042377 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-slash\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.042480 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.042417 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-var-lib-openvswitch\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.042480 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.042419 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.042589 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.042549 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.042879 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.042852 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 01 19:23:15.043002 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.042877 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e4f997c-ae5b-4400-874a-89c94b3a5796-konnectivity-ca\") pod \"konnectivity-agent-qvn4g\" (UID: \"2e4f997c-ae5b-4400-874a-89c94b3a5796\") " pod="kube-system/konnectivity-agent-qvn4g" Jul 01 19:23:15.043002 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.042982 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-ovnkube-config\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.043002 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.042991 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-systemd\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.043182 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043062 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-modprobe-d\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.043182 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.043093 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 19:23:15.043182 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043125 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-run-k8s-cni-cncf-io\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.043331 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043188 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-sysctl-d\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.043331 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.039958 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-run-ovn-kubernetes\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.043331 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043264 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-run-ovn\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.043472 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.043395 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs podName:688cb495-bdbd-40fc-a726-7bc07d269276 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:15.543350919 +0000 UTC m=+3.089727553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs") pod "network-metrics-daemon-m2flh" (UID: "688cb495-bdbd-40fc-a726-7bc07d269276") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 19:23:15.043472 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043448 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-etc-openvswitch\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.043586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043473 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04296533-7323-4246-bcf2-fc3cd1e387ef-serviceca\") pod \"node-ca-926nd\" (UID: \"04296533-7323-4246-bcf2-fc3cd1e387ef\") " pod="openshift-image-registry/node-ca-926nd" Jul 01 19:23:15.043586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043518 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-cni-netd\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.043586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043566 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-ovn-node-metrics-cert\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.043763 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043607 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-sysctl-conf\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.043763 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043646 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-registration-dir\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.043763 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043679 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-system-cni-dir\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.043763 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043720 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-var-lib-cni-multus\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.043763 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043758 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-run-netns\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.044002 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043795 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-cni-bin\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.044002 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043830 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-run\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.044002 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043842 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-ovnkube-script-lib\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.044002 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043859 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fba7d267-ad49-4729-8e5f-737cf617fcd6-iptables-alerter-script\") pod \"iptables-alerter-d6rls\" (UID: \"fba7d267-ad49-4729-8e5f-737cf617fcd6\") " pod="openshift-network-operator/iptables-alerter-d6rls" Jul 01 19:23:15.044002 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043864 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-socket-dir\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.044002 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.043954 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-sys-fs\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.044002 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044000 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-cni-netd\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.044357 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044011 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-cnibin\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.044357 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044048 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-cni-bin\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.044357 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044064 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-lib-modules\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.044357 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044119 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04296533-7323-4246-bcf2-fc3cd1e387ef-host\") pod \"node-ca-926nd\" (UID: \"04296533-7323-4246-bcf2-fc3cd1e387ef\") " pod="openshift-image-registry/node-ca-926nd" Jul 01 19:23:15.044357 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044123 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-sysconfig\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.044357 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044189 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-registration-dir\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.044357 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044198 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04296533-7323-4246-bcf2-fc3cd1e387ef-host\") pod \"node-ca-926nd\" (UID: \"04296533-7323-4246-bcf2-fc3cd1e387ef\") " pod="openshift-image-registry/node-ca-926nd" Jul 01 19:23:15.044357 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044226 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-run-openvswitch\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.044357 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044233 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-sys-fs\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.044795 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044362 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-socket-dir\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.044795 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044377 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-sysctl-conf\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.044795 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044427 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-run-netns\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.044795 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044429 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stprv\" (UniqueName: \"kubernetes.io/projected/688cb495-bdbd-40fc-a726-7bc07d269276-kube-api-access-stprv\") pod \"network-metrics-daemon-m2flh\" (UID: \"688cb495-bdbd-40fc-a726-7bc07d269276\") " pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:15.044795 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044507 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-kubelet\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.044795 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044524 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-run\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.044795 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044554 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-kubernetes\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.044795 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044568 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-run-systemd\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.044795 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044587 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/90fc2dc7-c463-49e5-b2f5-050c9238f37c-cni-binary-copy\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.044795 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044633 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-node-log\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.044795 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044718 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-node-log\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.044795 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044765 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ce6745e-2896-41de-b042-33adaac95142-etc-kubernetes\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.045382 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.044804 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-host-kubelet\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.045382 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.042856 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-env-overrides\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.048023 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.047997 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ce6745e-2896-41de-b042-33adaac95142-tmp\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.048161 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.048036 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ce6745e-2896-41de-b042-33adaac95142-etc-tuned\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.048327 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.048303 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e4f997c-ae5b-4400-874a-89c94b3a5796-agent-certs\") pod \"konnectivity-agent-qvn4g\" (UID: \"2e4f997c-ae5b-4400-874a-89c94b3a5796\") " pod="kube-system/konnectivity-agent-qvn4g" Jul 01 19:23:15.049219 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.049189 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-ovn-node-metrics-cert\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.054206 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.054179 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 01 19:23:15.054206 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.054208 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 01 19:23:15.054369 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.054222 2585 projected.go:196] Error preparing data for projected volume kube-api-access-wzpbv for pod openshift-network-diagnostics/network-check-target-csgsb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 19:23:15.054369 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.054292 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv podName:54990c60-95a2-4a2c-9d4d-690c9994707b nodeName:}" failed. No retries permitted until 2026-07-01 19:23:15.554274402 +0000 UTC m=+3.100651019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wzpbv" (UniqueName: "kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv") pod "network-check-target-csgsb" (UID: "54990c60-95a2-4a2c-9d4d-690c9994707b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 19:23:15.056642 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.056537 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlmld\" (UniqueName: \"kubernetes.io/projected/bfc13a1b-ac3d-4399-b42e-ebb80bba1dec-kube-api-access-jlmld\") pod \"aws-ebs-csi-driver-node-s6n9k\" (UID: \"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.057218 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.057194 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w7b5\" (UniqueName: \"kubernetes.io/projected/04296533-7323-4246-bcf2-fc3cd1e387ef-kube-api-access-4w7b5\") pod \"node-ca-926nd\" (UID: \"04296533-7323-4246-bcf2-fc3cd1e387ef\") " pod="openshift-image-registry/node-ca-926nd" Jul 01 19:23:15.057491 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.057460 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtk9j\" (UniqueName: \"kubernetes.io/projected/8c7695a8-08e8-4dc3-8c0f-121f69a158a4-kube-api-access-vtk9j\") pod \"ovnkube-node-frt9f\" (UID: \"8c7695a8-08e8-4dc3-8c0f-121f69a158a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.058251 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.058211 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdfch\" (UniqueName: \"kubernetes.io/projected/fba7d267-ad49-4729-8e5f-737cf617fcd6-kube-api-access-cdfch\") pod \"iptables-alerter-d6rls\" (UID: \"fba7d267-ad49-4729-8e5f-737cf617fcd6\") " pod="openshift-network-operator/iptables-alerter-d6rls" Jul 01 19:23:15.059361 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.059315 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dsmf\" (UniqueName: \"kubernetes.io/projected/7ce6745e-2896-41de-b042-33adaac95142-kube-api-access-4dsmf\") pod \"tuned-b9f2f\" (UID: \"7ce6745e-2896-41de-b042-33adaac95142\") " pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.060022 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.059991 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stprv\" (UniqueName: \"kubernetes.io/projected/688cb495-bdbd-40fc-a726-7bc07d269276-kube-api-access-stprv\") pod \"network-metrics-daemon-m2flh\" (UID: \"688cb495-bdbd-40fc-a726-7bc07d269276\") " pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:15.145680 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145595 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-var-lib-cni-bin\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.145680 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145628 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-var-lib-kubelet\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.145680 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145647 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e658cc5b-e0b2-4701-8568-5008df00ed5b-multus-daemon-config\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.145680 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145664 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-kubelet-config\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:15.145987 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145725 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-var-lib-cni-bin\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.145987 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145723 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-dbus\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:15.145987 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145791 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-var-lib-kubelet\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.145987 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145789 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:15.145987 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145826 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-hostroot\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.145987 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145857 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90fc2dc7-c463-49e5-b2f5-050c9238f37c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.145987 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145876 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/90fc2dc7-c463-49e5-b2f5-050c9238f37c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.145987 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145901 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjkph\" (UniqueName: \"kubernetes.io/projected/90fc2dc7-c463-49e5-b2f5-050c9238f37c-kube-api-access-vjkph\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.145987 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145928 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40a5624-aef1-4d71-a765-ccc60715a3d8-tmp-dir\") pod \"node-resolver-cgn6q\" (UID: \"a40a5624-aef1-4d71-a765-ccc60715a3d8\") " pod="openshift-dns/node-resolver-cgn6q" Jul 01 19:23:15.145987 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145946 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-hostroot\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.145987 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145963 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90fc2dc7-c463-49e5-b2f5-050c9238f37c-system-cni-dir\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.145987 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.145982 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 01 19:23:15.145987 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145996 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxxf\" (UniqueName: \"kubernetes.io/projected/a40a5624-aef1-4d71-a765-ccc60715a3d8-kube-api-access-zvxxf\") pod \"node-resolver-cgn6q\" (UID: \"a40a5624-aef1-4d71-a765-ccc60715a3d8\") " pod="openshift-dns/node-resolver-cgn6q" Jul 01 19:23:15.146649 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146024 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-os-release\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.146649 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.146045 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret podName:e2553301-dbf4-4594-9dea-5d6f3b6ff8f2 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:15.646022992 +0000 UTC m=+3.192399611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret") pod "global-pull-secret-syncer-qsrzd" (UID: "e2553301-dbf4-4594-9dea-5d6f3b6ff8f2") : object "kube-system"/"original-pull-secret" not registered Jul 01 19:23:15.146649 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146055 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90fc2dc7-c463-49e5-b2f5-050c9238f37c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.146649 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146090 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-os-release\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.146649 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146092 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-run-netns\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.146649 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.145963 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-dbus\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:15.146649 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146134 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-run-netns\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.146649 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146172 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-etc-kubernetes\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.146649 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146189 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-kubelet-config\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:15.146649 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146205 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-multus-cni-dir\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.146649 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146234 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-multus-socket-dir-parent\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.146649 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146261 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnmbx\" (UniqueName: \"kubernetes.io/projected/e658cc5b-e0b2-4701-8568-5008df00ed5b-kube-api-access-wnmbx\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.146649 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90fc2dc7-c463-49e5-b2f5-050c9238f37c-cnibin\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.146649 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146315 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-run-multus-certs\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146317 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-multus-cni-dir\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146322 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90fc2dc7-c463-49e5-b2f5-050c9238f37c-system-cni-dir\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146328 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e658cc5b-e0b2-4701-8568-5008df00ed5b-multus-daemon-config\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146327 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40a5624-aef1-4d71-a765-ccc60715a3d8-tmp-dir\") pod \"node-resolver-cgn6q\" (UID: \"a40a5624-aef1-4d71-a765-ccc60715a3d8\") " pod="openshift-dns/node-resolver-cgn6q" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146345 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-run-k8s-cni-cncf-io\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146381 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-multus-socket-dir-parent\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146386 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-run-k8s-cni-cncf-io\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146396 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-system-cni-dir\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146427 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-var-lib-cni-multus\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146459 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-cnibin\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146414 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-etc-kubernetes\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146459 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-run-multus-certs\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146492 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-host-var-lib-cni-multus\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146513 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90fc2dc7-c463-49e5-b2f5-050c9238f37c-cnibin\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.147326 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146536 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-cnibin\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147822 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146534 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/90fc2dc7-c463-49e5-b2f5-050c9238f37c-cni-binary-copy\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.147822 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146593 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/90fc2dc7-c463-49e5-b2f5-050c9238f37c-os-release\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.147822 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146546 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-system-cni-dir\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147822 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146618 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e658cc5b-e0b2-4701-8568-5008df00ed5b-cni-binary-copy\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147822 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146670 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/90fc2dc7-c463-49e5-b2f5-050c9238f37c-os-release\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.147822 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146680 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/90fc2dc7-c463-49e5-b2f5-050c9238f37c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.147822 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146708 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a40a5624-aef1-4d71-a765-ccc60715a3d8-hosts-file\") pod \"node-resolver-cgn6q\" (UID: \"a40a5624-aef1-4d71-a765-ccc60715a3d8\") " pod="openshift-dns/node-resolver-cgn6q" Jul 01 19:23:15.147822 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146730 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-multus-conf-dir\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147822 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146793 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e658cc5b-e0b2-4701-8568-5008df00ed5b-multus-conf-dir\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147822 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146802 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a40a5624-aef1-4d71-a765-ccc60715a3d8-hosts-file\") pod \"node-resolver-cgn6q\" (UID: \"a40a5624-aef1-4d71-a765-ccc60715a3d8\") " pod="openshift-dns/node-resolver-cgn6q" Jul 01 19:23:15.147822 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.146873 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/90fc2dc7-c463-49e5-b2f5-050c9238f37c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.147822 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.147077 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/90fc2dc7-c463-49e5-b2f5-050c9238f37c-cni-binary-copy\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.147822 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.147154 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e658cc5b-e0b2-4701-8568-5008df00ed5b-cni-binary-copy\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.147822 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.147183 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/90fc2dc7-c463-49e5-b2f5-050c9238f37c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.152954 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.152891 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:15.160590 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.160543 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnmbx\" (UniqueName: \"kubernetes.io/projected/e658cc5b-e0b2-4701-8568-5008df00ed5b-kube-api-access-wnmbx\") pod \"multus-zjd48\" (UID: \"e658cc5b-e0b2-4701-8568-5008df00ed5b\") " pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.160728 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.160671 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxxf\" (UniqueName: \"kubernetes.io/projected/a40a5624-aef1-4d71-a765-ccc60715a3d8-kube-api-access-zvxxf\") pod \"node-resolver-cgn6q\" (UID: \"a40a5624-aef1-4d71-a765-ccc60715a3d8\") " pod="openshift-dns/node-resolver-cgn6q" Jul 01 19:23:15.160817 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.160794 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjkph\" (UniqueName: \"kubernetes.io/projected/90fc2dc7-c463-49e5-b2f5-050c9238f37c-kube-api-access-vjkph\") pod \"multus-additional-cni-plugins-plcl5\" (UID: \"90fc2dc7-c463-49e5-b2f5-050c9238f37c\") " pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.161644 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.161626 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" Jul 01 19:23:15.171861 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.171832 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qvn4g" Jul 01 19:23:15.177856 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.177839 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-926nd" Jul 01 19:23:15.186748 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.186720 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d6rls" Jul 01 19:23:15.194811 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.194788 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" Jul 01 19:23:15.203692 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.203666 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-plcl5" Jul 01 19:23:15.212668 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.212643 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cgn6q" Jul 01 19:23:15.227443 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.227408 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zjd48" Jul 01 19:23:15.248103 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.248077 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 01 19:23:15.458603 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:15.458575 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04296533_7323_4246_bcf2_fc3cd1e387ef.slice/crio-50f8cd9261ba1b1d9a2c6a7a5881b6e60d0e96e48dac4cf2157c42bc074e7d74 WatchSource:0}: Error finding container 50f8cd9261ba1b1d9a2c6a7a5881b6e60d0e96e48dac4cf2157c42bc074e7d74: Status 404 returned error can't find the container with id 50f8cd9261ba1b1d9a2c6a7a5881b6e60d0e96e48dac4cf2157c42bc074e7d74 Jul 01 19:23:15.459574 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:15.459551 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4f997c_ae5b_4400_874a_89c94b3a5796.slice/crio-8a0ff3a1532ee196014cfe8a7ab27329107abfe6741716134f9a76f50331f700 WatchSource:0}: Error finding container 8a0ff3a1532ee196014cfe8a7ab27329107abfe6741716134f9a76f50331f700: Status 404 returned error can't find the container with id 8a0ff3a1532ee196014cfe8a7ab27329107abfe6741716134f9a76f50331f700 Jul 01 19:23:15.478595 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:15.478557 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc13a1b_ac3d_4399_b42e_ebb80bba1dec.slice/crio-24e18640629aaebca7b90091ab9f34364e12342457772b049e1d8bf838f1ba83 WatchSource:0}: Error finding container 24e18640629aaebca7b90091ab9f34364e12342457772b049e1d8bf838f1ba83: Status 404 returned error can't find the container with id 24e18640629aaebca7b90091ab9f34364e12342457772b049e1d8bf838f1ba83 Jul 01 19:23:15.479223 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:15.479199 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba7d267_ad49_4729_8e5f_737cf617fcd6.slice/crio-bf1811c6ca9c3fdf40f35eaba3eb1796e447187aa4677cf8ed7ca03887f2417b WatchSource:0}: Error finding container bf1811c6ca9c3fdf40f35eaba3eb1796e447187aa4677cf8ed7ca03887f2417b: Status 404 returned error can't find the container with id bf1811c6ca9c3fdf40f35eaba3eb1796e447187aa4677cf8ed7ca03887f2417b Jul 01 19:23:15.480573 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:15.480412 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode658cc5b_e0b2_4701_8568_5008df00ed5b.slice/crio-c33d2b7419a4182116f88671f8b215319b2218d971c0a514b20308e09d60316c WatchSource:0}: Error finding container c33d2b7419a4182116f88671f8b215319b2218d971c0a514b20308e09d60316c: Status 404 returned error can't find the container with id c33d2b7419a4182116f88671f8b215319b2218d971c0a514b20308e09d60316c Jul 01 19:23:15.481509 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:15.481472 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fc2dc7_c463_49e5_b2f5_050c9238f37c.slice/crio-aa5f3472338a0c8f78755c7c613e9e2708af2fcf97a05c76cf25085e3cd8e1f2 WatchSource:0}: Error finding container aa5f3472338a0c8f78755c7c613e9e2708af2fcf97a05c76cf25085e3cd8e1f2: Status 404 returned error can't find the container with id aa5f3472338a0c8f78755c7c613e9e2708af2fcf97a05c76cf25085e3cd8e1f2 Jul 01 19:23:15.482543 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:15.482490 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40a5624_aef1_4d71_a765_ccc60715a3d8.slice/crio-634d2ba997f323437e50bab81ee851862a94d7c2b256692df62f0eb9ae1eb4ba WatchSource:0}: Error finding container 634d2ba997f323437e50bab81ee851862a94d7c2b256692df62f0eb9ae1eb4ba: Status 404 returned error can't find the container with id 634d2ba997f323437e50bab81ee851862a94d7c2b256692df62f0eb9ae1eb4ba Jul 01 19:23:15.483595 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:15.483126 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7695a8_08e8_4dc3_8c0f_121f69a158a4.slice/crio-bb958f53476f05c4d66ad04f1827d581013a396c27e1bf779e1f3478cce4659b WatchSource:0}: Error finding container bb958f53476f05c4d66ad04f1827d581013a396c27e1bf779e1f3478cce4659b: Status 404 returned error can't find the container with id bb958f53476f05c4d66ad04f1827d581013a396c27e1bf779e1f3478cce4659b Jul 01 19:23:15.484697 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:15.484676 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce6745e_2896_41de_b042_33adaac95142.slice/crio-18e68414b24ce19c8c19e2c7872276c714a190ee054c7606bc19b2793409e7c2 WatchSource:0}: Error finding container 18e68414b24ce19c8c19e2c7872276c714a190ee054c7606bc19b2793409e7c2: Status 404 returned error can't find the container with id 18e68414b24ce19c8c19e2c7872276c714a190ee054c7606bc19b2793409e7c2 Jul 01 19:23:15.549871 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.549842 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs\") pod \"network-metrics-daemon-m2flh\" (UID: \"688cb495-bdbd-40fc-a726-7bc07d269276\") " pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:15.550033 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.550001 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 19:23:15.550112 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.550065 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs podName:688cb495-bdbd-40fc-a726-7bc07d269276 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:16.55004665 +0000 UTC m=+4.096423279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs") pod "network-metrics-daemon-m2flh" (UID: "688cb495-bdbd-40fc-a726-7bc07d269276") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 19:23:15.559504 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.559482 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 01 19:23:15.651067 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.651024 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpbv\" (UniqueName: \"kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv\") pod \"network-check-target-csgsb\" (UID: \"54990c60-95a2-4a2c-9d4d-690c9994707b\") " pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:15.651251 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.651117 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:15.651327 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.651248 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 01 19:23:15.651327 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.651246 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 01 19:23:15.651327 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.651273 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 01 19:23:15.651327 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.651287 2585 projected.go:196] Error preparing data for projected volume kube-api-access-wzpbv for pod openshift-network-diagnostics/network-check-target-csgsb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 19:23:15.651550 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.651338 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret podName:e2553301-dbf4-4594-9dea-5d6f3b6ff8f2 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:16.651317514 +0000 UTC m=+4.197694143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret") pod "global-pull-secret-syncer-qsrzd" (UID: "e2553301-dbf4-4594-9dea-5d6f3b6ff8f2") : object "kube-system"/"original-pull-secret" not registered Jul 01 19:23:15.651550 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:15.651359 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv podName:54990c60-95a2-4a2c-9d4d-690c9994707b nodeName:}" failed. No retries permitted until 2026-07-01 19:23:16.651347889 +0000 UTC m=+4.197724521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wzpbv" (UniqueName: "kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv") pod "network-check-target-csgsb" (UID: "54990c60-95a2-4a2c-9d4d-690c9994707b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 19:23:15.904534 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.904493 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 19:18:15 +0000 UTC" deadline="2028-01-23 09:25:02.229655627 +0000 UTC" Jul 01 19:23:15.904534 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.904529 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13694h1m46.325130321s" Jul 01 19:23:15.968437 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.968182 2585 generic.go:356] "Generic (PLEG): container finished" podID="c1a212c3b581e59189101d5b1344ee82" containerID="8cd94d554ccd80da03164e4f549c4f55fe36c9682dfd0435b10551e8030d3ff2" exitCode=0 Jul 01 19:23:15.968627 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.968268 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" event={"ID":"c1a212c3b581e59189101d5b1344ee82","Type":"ContainerDied","Data":"8cd94d554ccd80da03164e4f549c4f55fe36c9682dfd0435b10551e8030d3ff2"} Jul 01 19:23:15.970587 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.970519 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" event={"ID":"7ce6745e-2896-41de-b042-33adaac95142","Type":"ContainerStarted","Data":"18e68414b24ce19c8c19e2c7872276c714a190ee054c7606bc19b2793409e7c2"} Jul 01 19:23:15.972033 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.971956 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plcl5" event={"ID":"90fc2dc7-c463-49e5-b2f5-050c9238f37c","Type":"ContainerStarted","Data":"aa5f3472338a0c8f78755c7c613e9e2708af2fcf97a05c76cf25085e3cd8e1f2"} Jul 01 19:23:15.973678 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.973576 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d6rls" event={"ID":"fba7d267-ad49-4729-8e5f-737cf617fcd6","Type":"ContainerStarted","Data":"bf1811c6ca9c3fdf40f35eaba3eb1796e447187aa4677cf8ed7ca03887f2417b"} Jul 01 19:23:15.975082 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.975048 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zjd48" event={"ID":"e658cc5b-e0b2-4701-8568-5008df00ed5b","Type":"ContainerStarted","Data":"c33d2b7419a4182116f88671f8b215319b2218d971c0a514b20308e09d60316c"} Jul 01 19:23:15.976779 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.976756 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qvn4g" event={"ID":"2e4f997c-ae5b-4400-874a-89c94b3a5796","Type":"ContainerStarted","Data":"8a0ff3a1532ee196014cfe8a7ab27329107abfe6741716134f9a76f50331f700"} Jul 01 19:23:15.979129 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.979106 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" event={"ID":"8c7695a8-08e8-4dc3-8c0f-121f69a158a4","Type":"ContainerStarted","Data":"bb958f53476f05c4d66ad04f1827d581013a396c27e1bf779e1f3478cce4659b"} Jul 01 19:23:15.981241 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.981112 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cgn6q" event={"ID":"a40a5624-aef1-4d71-a765-ccc60715a3d8","Type":"ContainerStarted","Data":"634d2ba997f323437e50bab81ee851862a94d7c2b256692df62f0eb9ae1eb4ba"} Jul 01 19:23:15.985845 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.985753 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" event={"ID":"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec","Type":"ContainerStarted","Data":"24e18640629aaebca7b90091ab9f34364e12342457772b049e1d8bf838f1ba83"} Jul 01 19:23:15.987565 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:15.987533 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-926nd" event={"ID":"04296533-7323-4246-bcf2-fc3cd1e387ef","Type":"ContainerStarted","Data":"50f8cd9261ba1b1d9a2c6a7a5881b6e60d0e96e48dac4cf2157c42bc074e7d74"} Jul 01 19:23:16.559664 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:16.559615 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs\") pod \"network-metrics-daemon-m2flh\" (UID: \"688cb495-bdbd-40fc-a726-7bc07d269276\") " pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:16.559854 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:16.559802 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 19:23:16.559923 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:16.559887 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs podName:688cb495-bdbd-40fc-a726-7bc07d269276 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:18.559866118 +0000 UTC m=+6.106242750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs") pod "network-metrics-daemon-m2flh" (UID: "688cb495-bdbd-40fc-a726-7bc07d269276") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 19:23:16.660508 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:16.660463 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpbv\" (UniqueName: \"kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv\") pod \"network-check-target-csgsb\" (UID: \"54990c60-95a2-4a2c-9d4d-690c9994707b\") " pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:16.660680 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:16.660542 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:16.660680 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:16.660669 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 01 19:23:16.660820 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:16.660748 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret podName:e2553301-dbf4-4594-9dea-5d6f3b6ff8f2 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:18.660728923 +0000 UTC m=+6.207105543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret") pod "global-pull-secret-syncer-qsrzd" (UID: "e2553301-dbf4-4594-9dea-5d6f3b6ff8f2") : object "kube-system"/"original-pull-secret" not registered Jul 01 19:23:16.661260 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:16.661240 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 01 19:23:16.661332 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:16.661269 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 01 19:23:16.661332 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:16.661283 2585 projected.go:196] Error preparing data for projected volume kube-api-access-wzpbv for pod openshift-network-diagnostics/network-check-target-csgsb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 19:23:16.661422 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:16.661343 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv podName:54990c60-95a2-4a2c-9d4d-690c9994707b nodeName:}" failed. No retries permitted until 2026-07-01 19:23:18.661326715 +0000 UTC m=+6.207703346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wzpbv" (UniqueName: "kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv") pod "network-check-target-csgsb" (UID: "54990c60-95a2-4a2c-9d4d-690c9994707b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 19:23:16.905781 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:16.905652 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 19:18:15 +0000 UTC" deadline="2028-02-01 16:58:37.736077313 +0000 UTC" Jul 01 19:23:16.905781 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:16.905725 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13917h35m20.830356941s" Jul 01 19:23:16.960629 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:16.959813 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:16.960629 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:16.959958 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:16.960629 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:16.959813 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:16.960629 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:16.959971 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:16.960629 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:16.960118 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:16.960629 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:16.960205 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:16.995224 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:16.995182 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" event={"ID":"c1a212c3b581e59189101d5b1344ee82","Type":"ContainerStarted","Data":"88cec6dbd4573c98a679c6c33977da41eb6c2e781d93452db0259a6597c51028"} Jul 01 19:23:17.012898 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:17.012823 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" podStartSLOduration=3.012801528 podStartE2EDuration="3.012801528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:17.012221791 +0000 UTC m=+4.558598426" watchObservedRunningTime="2026-07-01 19:23:17.012801528 +0000 UTC m=+4.559178163" Jul 01 19:23:18.578722 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:18.578040 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs\") pod \"network-metrics-daemon-m2flh\" (UID: \"688cb495-bdbd-40fc-a726-7bc07d269276\") " pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:18.578722 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:18.578314 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 19:23:18.578722 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:18.578379 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs podName:688cb495-bdbd-40fc-a726-7bc07d269276 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:22.57836081 +0000 UTC m=+10.124737425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs") pod "network-metrics-daemon-m2flh" (UID: "688cb495-bdbd-40fc-a726-7bc07d269276") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 19:23:18.679080 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:18.679023 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:18.679283 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:18.679123 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpbv\" (UniqueName: \"kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv\") pod \"network-check-target-csgsb\" (UID: \"54990c60-95a2-4a2c-9d4d-690c9994707b\") " pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:18.679283 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:18.679275 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 01 19:23:18.679358 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:18.679294 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 01 19:23:18.679358 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:18.679309 2585 projected.go:196] Error preparing data for projected volume kube-api-access-wzpbv for pod openshift-network-diagnostics/network-check-target-csgsb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 19:23:18.679439 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:18.679374 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv podName:54990c60-95a2-4a2c-9d4d-690c9994707b nodeName:}" failed. No retries permitted until 2026-07-01 19:23:22.679356355 +0000 UTC m=+10.225732970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wzpbv" (UniqueName: "kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv") pod "network-check-target-csgsb" (UID: "54990c60-95a2-4a2c-9d4d-690c9994707b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 19:23:18.679778 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:18.679761 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 01 19:23:18.679840 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:18.679817 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret podName:e2553301-dbf4-4594-9dea-5d6f3b6ff8f2 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:22.679800295 +0000 UTC m=+10.226176906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret") pod "global-pull-secret-syncer-qsrzd" (UID: "e2553301-dbf4-4594-9dea-5d6f3b6ff8f2") : object "kube-system"/"original-pull-secret" not registered Jul 01 19:23:18.957877 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:18.957844 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:18.958085 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:18.957986 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:18.958382 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:18.958363 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:18.958558 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:18.958452 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:18.958558 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:18.958492 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:18.958662 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:18.958557 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:20.956360 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:20.956327 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:20.956360 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:20.956348 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:20.956860 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:20.956327 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:20.956860 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:20.956467 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:20.956860 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:20.956581 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:20.956860 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:20.956654 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:22.609662 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:22.609627 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs\") pod \"network-metrics-daemon-m2flh\" (UID: \"688cb495-bdbd-40fc-a726-7bc07d269276\") " pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:22.610136 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:22.609770 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 19:23:22.610136 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:22.609836 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs podName:688cb495-bdbd-40fc-a726-7bc07d269276 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:30.609817714 +0000 UTC m=+18.156194331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs") pod "network-metrics-daemon-m2flh" (UID: "688cb495-bdbd-40fc-a726-7bc07d269276") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 19:23:22.710306 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:22.710269 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpbv\" (UniqueName: \"kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv\") pod \"network-check-target-csgsb\" (UID: \"54990c60-95a2-4a2c-9d4d-690c9994707b\") " pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:22.710504 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:22.710339 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:22.710504 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:22.710449 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 01 19:23:22.710504 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:22.710453 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 01 19:23:22.710504 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:22.710478 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 01 19:23:22.710504 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:22.710492 2585 projected.go:196] Error preparing data for projected volume kube-api-access-wzpbv for pod openshift-network-diagnostics/network-check-target-csgsb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 19:23:22.710739 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:22.710510 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret podName:e2553301-dbf4-4594-9dea-5d6f3b6ff8f2 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:30.710491614 +0000 UTC m=+18.256868229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret") pod "global-pull-secret-syncer-qsrzd" (UID: "e2553301-dbf4-4594-9dea-5d6f3b6ff8f2") : object "kube-system"/"original-pull-secret" not registered Jul 01 19:23:22.710739 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:22.710549 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv podName:54990c60-95a2-4a2c-9d4d-690c9994707b nodeName:}" failed. No retries permitted until 2026-07-01 19:23:30.710531782 +0000 UTC m=+18.256908417 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wzpbv" (UniqueName: "kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv") pod "network-check-target-csgsb" (UID: "54990c60-95a2-4a2c-9d4d-690c9994707b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 19:23:22.956765 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:22.956676 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:22.956914 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:22.956802 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:22.957206 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:22.957186 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:22.957304 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:22.957285 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:22.957353 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:22.957338 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:22.957438 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:22.957405 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:24.956328 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:24.956227 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:24.956328 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:24.956220 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:24.956328 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:24.956227 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:24.956838 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:24.956352 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:24.956838 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:24.956422 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:24.956838 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:24.956520 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:26.956356 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:26.956285 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:26.956848 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:26.956285 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:26.956848 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:26.956427 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:26.956848 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:26.956566 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:26.956848 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:26.956294 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:26.956848 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:26.956654 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:28.955983 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:28.955952 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:28.956361 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:28.955952 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:28.956361 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:28.956092 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:28.956361 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:28.956080 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:28.956361 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:28.956190 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:28.956361 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:28.956262 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:30.023907 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.023562 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cgn6q" event={"ID":"a40a5624-aef1-4d71-a765-ccc60715a3d8","Type":"ContainerStarted","Data":"ceff3b113959b33b2faf235cb583e829ac61bfb8df1f957a90f732b1f6484537"} Jul 01 19:23:30.025445 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.025407 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" event={"ID":"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec","Type":"ContainerStarted","Data":"71771696c02fab7e7b4f0abf5040ab4daf95d80636e4efdb302d45555e7853bc"} Jul 01 19:23:30.027154 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.026921 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-926nd" event={"ID":"04296533-7323-4246-bcf2-fc3cd1e387ef","Type":"ContainerStarted","Data":"1a387f4eb9542e138970a71046cf0679745d4212deec96c1aaf0cb6ed3e7f108"} Jul 01 19:23:30.028575 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.028545 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" event={"ID":"0ad6b0a84840892a426fb300e0c2d8d6","Type":"ContainerStarted","Data":"66237bf84bea8f6c4c3424ca45fb8f7aa6d32abd03e298d34fd0ba6b79eeae97"} Jul 01 19:23:30.030022 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.029990 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" event={"ID":"7ce6745e-2896-41de-b042-33adaac95142","Type":"ContainerStarted","Data":"84bf80935383938e6c8ab8d899e59b631fcd1721ef422409be20068548a6a801"} Jul 01 19:23:30.031751 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.031727 2585 generic.go:356] "Generic (PLEG): container finished" podID="90fc2dc7-c463-49e5-b2f5-050c9238f37c" containerID="6adc0337c93212c8a461f9e995b4067ca4f3a201a12b4fffe18f9f4e0e1af3c4" exitCode=0 Jul 01 19:23:30.031861 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.031811 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plcl5" event={"ID":"90fc2dc7-c463-49e5-b2f5-050c9238f37c","Type":"ContainerDied","Data":"6adc0337c93212c8a461f9e995b4067ca4f3a201a12b4fffe18f9f4e0e1af3c4"} Jul 01 19:23:30.033993 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.033969 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zjd48" event={"ID":"e658cc5b-e0b2-4701-8568-5008df00ed5b","Type":"ContainerStarted","Data":"4033e4c3da05544e8fb1f1dee9288fd12ed754cbe36f987c577caeaeb7befc34"} Jul 01 19:23:30.035459 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.035438 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qvn4g" event={"ID":"2e4f997c-ae5b-4400-874a-89c94b3a5796","Type":"ContainerStarted","Data":"1aca10fdf7405c79217fedb301fea945d58b089822a39f5ed1004602dea8e196"} Jul 01 19:23:30.038799 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.038755 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cgn6q" podStartSLOduration=3.682506265 podStartE2EDuration="17.038741079s" totalImagesPullingTime="13.356234814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:30.038570379 +0000 UTC m=+17.584947014" watchObservedRunningTime="2026-07-01 19:23:30.038741079 +0000 UTC m=+17.585117713" Jul 01 19:23:30.053805 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.053726 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qvn4g" podStartSLOduration=8.60624051 podStartE2EDuration="17.053706814s" totalImagesPullingTime="8.447466304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:30.053230727 +0000 UTC m=+17.599607361" watchObservedRunningTime="2026-07-01 19:23:30.053706814 +0000 UTC m=+17.600083449" Jul 01 19:23:30.067829 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.067778 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-926nd" podStartSLOduration=3.330692052 podStartE2EDuration="17.067767673s" totalImagesPullingTime="13.737075621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:30.067552452 +0000 UTC m=+17.613929088" watchObservedRunningTime="2026-07-01 19:23:30.067767673 +0000 UTC m=+17.614144309" Jul 01 19:23:30.108901 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.108843 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" podStartSLOduration=16.108823531 podStartE2EDuration="16.108823531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:30.108567877 +0000 UTC m=+17.654944510" watchObservedRunningTime="2026-07-01 19:23:30.108823531 +0000 UTC m=+17.655200167" Jul 01 19:23:30.127202 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.127149 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b9f2f" podStartSLOduration=3.360249386 podStartE2EDuration="17.12713046s" totalImagesPullingTime="13.766881074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:30.126540205 +0000 UTC m=+17.672916840" watchObservedRunningTime="2026-07-01 19:23:30.12713046 +0000 UTC m=+17.673507094" Jul 01 19:23:30.145823 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.145767 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zjd48" podStartSLOduration=3.323724286 podStartE2EDuration="17.145745594s" totalImagesPullingTime="13.822021308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:30.14446456 +0000 UTC m=+17.690841207" watchObservedRunningTime="2026-07-01 19:23:30.145745594 +0000 UTC m=+17.692122230" Jul 01 19:23:30.675441 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.675404 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs\") pod \"network-metrics-daemon-m2flh\" (UID: \"688cb495-bdbd-40fc-a726-7bc07d269276\") " pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:30.675620 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:30.675575 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 19:23:30.675660 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:30.675640 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs podName:688cb495-bdbd-40fc-a726-7bc07d269276 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:46.675624567 +0000 UTC m=+34.222001182 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs") pod "network-metrics-daemon-m2flh" (UID: "688cb495-bdbd-40fc-a726-7bc07d269276") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 19:23:30.775885 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.775853 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpbv\" (UniqueName: \"kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv\") pod \"network-check-target-csgsb\" (UID: \"54990c60-95a2-4a2c-9d4d-690c9994707b\") " pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:30.776091 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.775909 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:30.776091 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:30.776006 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 01 19:23:30.776091 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:30.776023 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 01 19:23:30.776091 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:30.776044 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 01 19:23:30.776091 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:30.776057 2585 projected.go:196] Error preparing data for projected volume kube-api-access-wzpbv for pod openshift-network-diagnostics/network-check-target-csgsb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 19:23:30.776091 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:30.776084 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret podName:e2553301-dbf4-4594-9dea-5d6f3b6ff8f2 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:46.776052675 +0000 UTC m=+34.322429302 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret") pod "global-pull-secret-syncer-qsrzd" (UID: "e2553301-dbf4-4594-9dea-5d6f3b6ff8f2") : object "kube-system"/"original-pull-secret" not registered Jul 01 19:23:30.776350 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:30.776114 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv podName:54990c60-95a2-4a2c-9d4d-690c9994707b nodeName:}" failed. No retries permitted until 2026-07-01 19:23:46.776103013 +0000 UTC m=+34.322479639 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wzpbv" (UniqueName: "kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv") pod "network-check-target-csgsb" (UID: "54990c60-95a2-4a2c-9d4d-690c9994707b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 19:23:30.955685 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.955605 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:30.955685 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.955626 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:30.956036 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:30.955613 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:30.959288 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:30.959252 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:30.959288 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:30.959267 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:30.959988 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:30.959948 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:31.039164 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:31.039124 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d6rls" event={"ID":"fba7d267-ad49-4729-8e5f-737cf617fcd6","Type":"ContainerStarted","Data":"7984ef1acf6456b659a78528b72d6698c946e2f86dcbc486a83d9e212063057f"} Jul 01 19:23:31.059119 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:31.059054 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d6rls" podStartSLOduration=8.85023932 podStartE2EDuration="18.059037263s" totalImagesPullingTime="9.208797943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:31.058446829 +0000 UTC m=+18.604823462" watchObservedRunningTime="2026-07-01 19:23:31.059037263 +0000 UTC m=+18.605413896" Jul 01 19:23:32.956338 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:32.956263 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:32.956891 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:32.956369 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:32.956891 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:32.956386 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:32.956891 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:32.956498 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:32.956891 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:32.956391 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:32.956891 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:32.956566 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:34.538045 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:34.537854 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qvn4g" Jul 01 19:23:34.538683 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:34.538664 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qvn4g" Jul 01 19:23:34.618576 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:34.618544 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 01 19:23:34.943997 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:34.943691 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T19:23:34.618561721Z","UUID":"ef7813a6-68f4-456a-8c4f-4f2ceb3d05e2","Handler":null,"Name":"","Endpoint":""} Jul 01 19:23:34.945230 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:34.945213 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 01 19:23:34.945331 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:34.945248 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 01 19:23:34.956256 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:34.956237 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:34.956352 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:34.956236 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:34.956418 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:34.956345 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:34.956475 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:34.956426 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:34.956475 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:34.956241 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:34.956549 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:34.956497 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:35.052620 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:35.052535 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" event={"ID":"8c7695a8-08e8-4dc3-8c0f-121f69a158a4","Type":"ContainerStarted","Data":"fd89fa2633ec5df7f361c64aa04bfb36e7f1101087489ad2b9fc7073165e8952"} Jul 01 19:23:35.052785 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:35.052636 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" event={"ID":"8c7695a8-08e8-4dc3-8c0f-121f69a158a4","Type":"ContainerStarted","Data":"4d343603fd63852099517c1b506ca60affa41ca3942013a5381d21b5929bcae3"} Jul 01 19:23:35.052785 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:35.052687 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" event={"ID":"8c7695a8-08e8-4dc3-8c0f-121f69a158a4","Type":"ContainerStarted","Data":"d6dd3211a5a6b33504adbb4af6c0e5fd6d02633de4907e9709b3a863c17f7b58"} Jul 01 19:23:35.052785 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:35.052733 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" event={"ID":"8c7695a8-08e8-4dc3-8c0f-121f69a158a4","Type":"ContainerStarted","Data":"09d804c26469340ef3f02d7232ec3efaca006e96618b1c96d39c2228ad31c73c"} Jul 01 19:23:35.052785 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:35.052777 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" event={"ID":"8c7695a8-08e8-4dc3-8c0f-121f69a158a4","Type":"ContainerStarted","Data":"26e1200f61d90befef1e04fcd9b941a3bdc842d6c6ecd5bf0ede82099945b944"} Jul 01 19:23:35.052982 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:35.052840 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" event={"ID":"8c7695a8-08e8-4dc3-8c0f-121f69a158a4","Type":"ContainerStarted","Data":"dcfd299bbc7c35ff751e3e2db4f5a87d2f894f28ce3228790f8a6d49b6ef8a54"} Jul 01 19:23:35.054798 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:35.054765 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" event={"ID":"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec","Type":"ContainerStarted","Data":"3cac4918ba3aa37c3bf253c51a9bcf12bda0096d3ab5a4358c25d82e100fe2c4"} Jul 01 19:23:35.055054 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:35.055008 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qvn4g" Jul 01 19:23:35.055535 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:35.055487 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qvn4g" Jul 01 19:23:36.059497 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:36.059400 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" event={"ID":"bfc13a1b-ac3d-4399-b42e-ebb80bba1dec","Type":"ContainerStarted","Data":"ec788e09d66bdec2a71ad94ddbe69444a9db887da571a1896477542c0477cb5b"} Jul 01 19:23:36.080200 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:36.080142 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6n9k" podStartSLOduration=3.232452815 podStartE2EDuration="23.08012396s" totalImagesPullingTime="19.847671145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:23:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:36.079904936 +0000 UTC m=+23.626281574" watchObservedRunningTime="2026-07-01 19:23:36.08012396 +0000 UTC m=+23.626500583" Jul 01 19:23:36.956381 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:36.956348 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:36.956568 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:36.956356 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:36.956568 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:36.956486 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:36.956568 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:36.956356 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:36.956728 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:36.956537 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:36.956728 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:36.956646 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:38.956041 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:38.956009 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:38.956561 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:38.956009 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:38.956561 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:38.956151 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:38.956561 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:38.956015 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:38.956561 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:38.956182 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:38.956561 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:38.956268 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:39.068701 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:39.068660 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" event={"ID":"8c7695a8-08e8-4dc3-8c0f-121f69a158a4","Type":"ContainerStarted","Data":"3077ff17b7a3b09cbf51c1b5c29636200ca126eb5b60d74c5f03e3017486712c"} Jul 01 19:23:39.070220 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:39.070197 2585 generic.go:356] "Generic (PLEG): container finished" podID="90fc2dc7-c463-49e5-b2f5-050c9238f37c" containerID="ea0b5009f28c1f4ab6d71af40e1f3d4286ac6cd5842db36eb6e3124e3b54be83" exitCode=0 Jul 01 19:23:39.070318 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:39.070230 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plcl5" event={"ID":"90fc2dc7-c463-49e5-b2f5-050c9238f37c","Type":"ContainerDied","Data":"ea0b5009f28c1f4ab6d71af40e1f3d4286ac6cd5842db36eb6e3124e3b54be83"} Jul 01 19:23:40.074613 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:40.074578 2585 generic.go:356] "Generic (PLEG): container finished" podID="90fc2dc7-c463-49e5-b2f5-050c9238f37c" containerID="0e5c04cfa33f5725696dc353879e29696302d9d3cc97c21af239d54d343212e5" exitCode=0 Jul 01 19:23:40.075211 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:40.074636 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plcl5" event={"ID":"90fc2dc7-c463-49e5-b2f5-050c9238f37c","Type":"ContainerDied","Data":"0e5c04cfa33f5725696dc353879e29696302d9d3cc97c21af239d54d343212e5"} Jul 01 19:23:40.956639 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:40.956406 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:40.956780 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:40.956406 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:40.956780 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:40.956683 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:40.956780 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:40.956734 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:40.956780 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:40.956456 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:40.956921 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:40.956799 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:41.078564 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:41.078532 2585 generic.go:356] "Generic (PLEG): container finished" podID="90fc2dc7-c463-49e5-b2f5-050c9238f37c" containerID="658d43f1119ea1dc4d1208a768aca4ca4837a6c605ebdeb822da366fd700fcad" exitCode=0 Jul 01 19:23:41.079152 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:41.078617 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plcl5" event={"ID":"90fc2dc7-c463-49e5-b2f5-050c9238f37c","Type":"ContainerDied","Data":"658d43f1119ea1dc4d1208a768aca4ca4837a6c605ebdeb822da366fd700fcad"} Jul 01 19:23:41.082397 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:41.082376 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" event={"ID":"8c7695a8-08e8-4dc3-8c0f-121f69a158a4","Type":"ContainerStarted","Data":"a846d89bf5c5ca6cbf1c74d8b83f2b882215deb61d0179214fc46788a463ab56"} Jul 01 19:23:41.082631 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:41.082615 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:41.082755 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:41.082742 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:41.082816 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:41.082795 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:41.098829 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:41.098796 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:41.098944 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:41.098896 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:23:41.130689 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:41.130638 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" podStartSLOduration=9.052123654 podStartE2EDuration="28.130624206s" totalImagesPullingTime="19.078500552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:41.130303698 +0000 UTC m=+28.676680332" watchObservedRunningTime="2026-07-01 19:23:41.130624206 +0000 UTC m=+28.677000839" Jul 01 19:23:42.772220 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:42.772133 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-csgsb"] Jul 01 19:23:42.772714 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:42.772265 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:42.772714 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:42.772370 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:42.775670 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:42.775617 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qsrzd"] Jul 01 19:23:42.776055 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:42.775912 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:42.776055 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:42.776013 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:42.776742 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:42.776707 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m2flh"] Jul 01 19:23:42.776888 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:42.776803 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:42.776953 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:42.776885 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:43.955843 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:43.955624 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:43.956283 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:43.955944 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:44.956330 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:44.956293 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:44.956708 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:44.956345 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:44.956708 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:44.956440 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:44.956708 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:44.956529 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:45.956274 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:45.956239 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:45.956583 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:45.956353 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgsb" podUID="54990c60-95a2-4a2c-9d4d-690c9994707b" Jul 01 19:23:46.689980 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:46.689895 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs\") pod \"network-metrics-daemon-m2flh\" (UID: \"688cb495-bdbd-40fc-a726-7bc07d269276\") " pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:46.690137 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:46.690022 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 19:23:46.690137 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:46.690108 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs podName:688cb495-bdbd-40fc-a726-7bc07d269276 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:18.690091902 +0000 UTC m=+66.236468519 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs") pod "network-metrics-daemon-m2flh" (UID: "688cb495-bdbd-40fc-a726-7bc07d269276") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 19:23:46.790214 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:46.790175 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpbv\" (UniqueName: \"kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv\") pod \"network-check-target-csgsb\" (UID: \"54990c60-95a2-4a2c-9d4d-690c9994707b\") " pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:46.790380 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:46.790230 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:46.790380 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:46.790316 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 01 19:23:46.790380 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:46.790329 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 01 19:23:46.790380 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:46.790344 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 01 19:23:46.790380 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:46.790356 2585 projected.go:196] Error preparing data for projected volume kube-api-access-wzpbv for pod openshift-network-diagnostics/network-check-target-csgsb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 19:23:46.790550 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:46.790381 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret podName:e2553301-dbf4-4594-9dea-5d6f3b6ff8f2 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:18.790363646 +0000 UTC m=+66.336740271 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret") pod "global-pull-secret-syncer-qsrzd" (UID: "e2553301-dbf4-4594-9dea-5d6f3b6ff8f2") : object "kube-system"/"original-pull-secret" not registered Jul 01 19:23:46.790550 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:46.790401 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv podName:54990c60-95a2-4a2c-9d4d-690c9994707b nodeName:}" failed. No retries permitted until 2026-07-01 19:24:18.790393047 +0000 UTC m=+66.336769663 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wzpbv" (UniqueName: "kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv") pod "network-check-target-csgsb" (UID: "54990c60-95a2-4a2c-9d4d-690c9994707b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 19:23:46.958489 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:46.958429 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:46.958940 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:46.958534 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2flh" podUID="688cb495-bdbd-40fc-a726-7bc07d269276" Jul 01 19:23:46.958940 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:46.958582 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:46.958940 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:46.958710 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsrzd" podUID="e2553301-dbf4-4594-9dea-5d6f3b6ff8f2" Jul 01 19:23:47.099607 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.099575 2585 generic.go:356] "Generic (PLEG): container finished" podID="90fc2dc7-c463-49e5-b2f5-050c9238f37c" containerID="d9cf99c248aa8a4adbd1fe60fe3b4e54f49ebabe1237140bf8e969746766195e" exitCode=0 Jul 01 19:23:47.099790 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.099626 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plcl5" event={"ID":"90fc2dc7-c463-49e5-b2f5-050c9238f37c","Type":"ContainerDied","Data":"d9cf99c248aa8a4adbd1fe60fe3b4e54f49ebabe1237140bf8e969746766195e"} Jul 01 19:23:47.193013 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.192983 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeReady" Jul 01 19:23:47.193190 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.193139 2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 01 19:23:47.233869 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.233787 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf"] Jul 01 19:23:47.247571 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.247545 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d9b4f4577-nvd9d"] Jul 01 19:23:47.247701 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.247679 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" Jul 01 19:23:47.252423 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.252400 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 01 19:23:47.252622 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.252598 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 01 19:23:47.253102 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.253079 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 01 19:23:47.255937 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.255797 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 01 19:23:47.255937 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.255809 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wxj8k\"" Jul 01 19:23:47.261806 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.261781 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w"] Jul 01 19:23:47.261937 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.261923 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.265936 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.265916 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 01 19:23:47.266121 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.266102 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nzzx8\"" Jul 01 19:23:47.267022 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.266977 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 01 19:23:47.267281 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.267137 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 01 19:23:47.278156 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.278119 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5f67b55555-428zv"] Jul 01 19:23:47.279762 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.279741 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" Jul 01 19:23:47.289224 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.289200 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 01 19:23:47.289349 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.289291 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 01 19:23:47.289349 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.289295 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lm6s9\"" Jul 01 19:23:47.290238 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.290219 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-x7hw2"] Jul 01 19:23:47.290353 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.290337 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.292637 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.292614 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 01 19:23:47.293137 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.292906 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-c6r68\"" Jul 01 19:23:47.293137 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.292946 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 01 19:23:47.293456 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.293365 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/44087f53-56ca-43d7-aaf8-582ee3ba0c1c-config\") pod \"service-ca-operator-5d7b7dc95c-kmnlf\" (UID: \"44087f53-56ca-43d7-aaf8-582ee3ba0c1c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" Jul 01 19:23:47.293456 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.293401 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbzth\" (UniqueName: \"kubernetes.io/projected/44087f53-56ca-43d7-aaf8-582ee3ba0c1c-kube-api-access-kbzth\") pod \"service-ca-operator-5d7b7dc95c-kmnlf\" (UID: \"44087f53-56ca-43d7-aaf8-582ee3ba0c1c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" Jul 01 19:23:47.293587 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.293488 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44087f53-56ca-43d7-aaf8-582ee3ba0c1c-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-kmnlf\" (UID: \"44087f53-56ca-43d7-aaf8-582ee3ba0c1c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" Jul 01 19:23:47.294737 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.294665 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 01 19:23:47.295225 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.295088 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 01 19:23:47.296118 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.295524 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 01 19:23:47.296118 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.295533 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 01 19:23:47.300366 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.300348 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6fb847847c-xqccn"] Jul 01 19:23:47.300564 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.300551 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-x7hw2" Jul 01 19:23:47.301520 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.301505 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 01 19:23:47.303364 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.303337 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-llzfn\"" Jul 01 19:23:47.303479 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.303384 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 01 19:23:47.303479 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.303409 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 01 19:23:47.312442 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.312416 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-m82rf"] Jul 01 19:23:47.312559 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.312500 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:47.315671 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.315650 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 01 19:23:47.315797 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.315679 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 01 19:23:47.315797 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.315693 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 01 19:23:47.315996 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.315974 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 01 19:23:47.316126 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.316028 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tqsg9\"" Jul 01 19:23:47.316126 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.316098 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 01 19:23:47.316243 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.315974 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 01 19:23:47.324445 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.324427 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf"] Jul 01 19:23:47.324550 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.324455 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-m82rf"] Jul 01 19:23:47.324550 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.324469 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-428zv"] Jul 01 19:23:47.324550 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.324481 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-x7hw2"] Jul 01 19:23:47.324550 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.324499 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j8kx5"] Jul 01 19:23:47.324550 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.324540 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:47.326796 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.326779 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 01 19:23:47.326894 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.326805 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 01 19:23:47.326894 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.326880 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-99n4q\"" Jul 01 19:23:47.327135 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.327118 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 01 19:23:47.327268 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.327254 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 01 19:23:47.331284 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.331267 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 01 19:23:47.338642 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.338624 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5"] Jul 01 19:23:47.338742 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.338730 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:47.341420 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.341401 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 01 19:23:47.341496 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.341420 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9p2w7\"" Jul 01 19:23:47.341651 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.341639 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 01 19:23:47.351509 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.351489 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-sn7px"] Jul 01 19:23:47.351605 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.351591 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:23:47.353926 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.353899 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:47.354023 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.353966 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 01 19:23:47.354511 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.354271 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 01 19:23:47.355699 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.354888 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gt8f8\"" Jul 01 19:23:47.355871 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.355851 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 01 19:23:47.356095 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.356058 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 01 19:23:47.365687 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.365665 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf"] Jul 01 19:23:47.366129 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.365749 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-sn7px" Jul 01 19:23:47.367780 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.367764 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 01 19:23:47.367872 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.367855 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 01 19:23:47.368060 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.368046 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hjpd5\"" Jul 01 19:23:47.379706 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.379687 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z82nd"] Jul 01 19:23:47.379817 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.379804 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" Jul 01 19:23:47.383380 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.383359 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 01 19:23:47.383380 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.383375 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 01 19:23:47.383528 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.383422 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 01 19:23:47.383528 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.383428 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 01 19:23:47.383627 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.383563 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-lwmhd\"" Jul 01 19:23:47.383827 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.383809 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr"] Jul 01 19:23:47.383951 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.383936 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z82nd" Jul 01 19:23:47.385963 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.385946 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 01 19:23:47.386051 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.386012 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qjjt5\"" Jul 01 19:23:47.386558 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.386544 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 01 19:23:47.388173 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.388028 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w"] Jul 01 19:23:47.388173 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.388116 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d9b4f4577-nvd9d"] Jul 01 19:23:47.388173 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.388165 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 01 19:23:47.388333 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.388201 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" Jul 01 19:23:47.388333 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.388246 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf"] Jul 01 19:23:47.388333 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.388269 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j8kx5"] Jul 01 19:23:47.388333 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.388294 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z82nd"] Jul 01 19:23:47.388333 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.388315 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-sn7px"] Jul 01 19:23:47.388333 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.388334 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr"] Jul 01 19:23:47.388553 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.388353 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6fb847847c-xqccn"] Jul 01 19:23:47.388553 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.388371 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5"] Jul 01 19:23:47.390340 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.390320 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 01 19:23:47.390598 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.390585 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-zcpfc\"" Jul 01 19:23:47.391026 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.391011 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 01 19:23:47.394097 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394076 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blswp\" (UniqueName: \"kubernetes.io/projected/5168c17b-70f2-4e40-ba13-d73744c17e7a-kube-api-access-blswp\") pod \"cluster-samples-operator-9ddc5ff76-2n74w\" (UID: \"5168c17b-70f2-4e40-ba13-d73744c17e7a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" Jul 01 19:23:47.394168 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394114 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94xpg\" (UniqueName: \"kubernetes.io/projected/d93cdf36-ab87-4ac4-b4a3-5ed00053c479-kube-api-access-94xpg\") pod \"volume-data-source-validator-69d9f679db-x7hw2\" (UID: \"d93cdf36-ab87-4ac4-b4a3-5ed00053c479\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-x7hw2" Jul 01 19:23:47.394168 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394135 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-serving-cert\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.394168 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394157 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbzth\" (UniqueName: \"kubernetes.io/projected/44087f53-56ca-43d7-aaf8-582ee3ba0c1c-kube-api-access-kbzth\") pod \"service-ca-operator-5d7b7dc95c-kmnlf\" (UID: \"44087f53-56ca-43d7-aaf8-582ee3ba0c1c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" Jul 01 19:23:47.394290 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394181 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-tmp\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.394290 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394237 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7hfj\" (UniqueName: \"kubernetes.io/projected/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-kube-api-access-b7hfj\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.394290 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394277 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/94519212-92c3-47b1-98d5-a1dfff195e48-image-registry-private-configuration\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.394379 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394309 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2jhr\" (UniqueName: \"kubernetes.io/projected/0c94f978-1520-4f79-91df-f1f0d8048877-kube-api-access-q2jhr\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:47.394423 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394375 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-default-certificate\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:47.394423 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394412 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:47.394498 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394462 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e53cdc86-b072-4c1e-9ca3-3d504c0bc6af-trusted-ca\") pod \"console-operator-7fdbdf9c99-m82rf\" (UID: \"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:47.394498 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394482 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:47.394555 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394497 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-bound-sa-token\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.394555 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394520 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c57q8\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-kube-api-access-c57q8\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.394555 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394542 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:23:47.394646 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394558 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/94519212-92c3-47b1-98d5-a1dfff195e48-ca-trust-extracted\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.394646 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394576 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-2n74w\" (UID: \"5168c17b-70f2-4e40-ba13-d73744c17e7a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" Jul 01 19:23:47.394646 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394601 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ef688413-3e26-4dfd-87c9-059133228994-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:23:47.394729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394649 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94519212-92c3-47b1-98d5-a1dfff195e48-trusted-ca\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.394729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394677 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/44087f53-56ca-43d7-aaf8-582ee3ba0c1c-config\") pod \"service-ca-operator-5d7b7dc95c-kmnlf\" (UID: \"44087f53-56ca-43d7-aaf8-582ee3ba0c1c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" Jul 01 19:23:47.394729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394701 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9d8r\" (UniqueName: \"kubernetes.io/projected/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-kube-api-access-d9d8r\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:47.394831 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394725 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e53cdc86-b072-4c1e-9ca3-3d504c0bc6af-config\") pod \"console-operator-7fdbdf9c99-m82rf\" (UID: \"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:47.394831 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394748 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-config-volume\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:47.394901 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394856 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.394901 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394884 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/94519212-92c3-47b1-98d5-a1dfff195e48-registry-certificates\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.394966 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394913 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.394966 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394940 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-service-ca-bundle\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.395056 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.394979 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/94519212-92c3-47b1-98d5-a1dfff195e48-installation-pull-secrets\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.395224 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.395166 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-snapshots\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.395224 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.395197 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:47.395324 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.395238 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44087f53-56ca-43d7-aaf8-582ee3ba0c1c-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-kmnlf\" (UID: \"44087f53-56ca-43d7-aaf8-582ee3ba0c1c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" Jul 01 19:23:47.395324 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.395264 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-stats-auth\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:47.395324 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.395291 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e53cdc86-b072-4c1e-9ca3-3d504c0bc6af-serving-cert\") pod \"console-operator-7fdbdf9c99-m82rf\" (UID: \"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:47.395485 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.395352 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmlhr\" (UniqueName: \"kubernetes.io/projected/ef688413-3e26-4dfd-87c9-059133228994-kube-api-access-rmlhr\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:23:47.395485 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.395415 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84j7l\" (UniqueName: \"kubernetes.io/projected/e53cdc86-b072-4c1e-9ca3-3d504c0bc6af-kube-api-access-84j7l\") pod \"console-operator-7fdbdf9c99-m82rf\" (UID: \"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:47.395485 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.395440 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-tmp-dir\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:47.395808 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.395792 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/44087f53-56ca-43d7-aaf8-582ee3ba0c1c-config\") pod \"service-ca-operator-5d7b7dc95c-kmnlf\" (UID: \"44087f53-56ca-43d7-aaf8-582ee3ba0c1c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" Jul 01 19:23:47.399012 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.398996 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44087f53-56ca-43d7-aaf8-582ee3ba0c1c-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-kmnlf\" (UID: \"44087f53-56ca-43d7-aaf8-582ee3ba0c1c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" Jul 01 19:23:47.408404 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.408386 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbzth\" (UniqueName: \"kubernetes.io/projected/44087f53-56ca-43d7-aaf8-582ee3ba0c1c-kube-api-access-kbzth\") pod \"service-ca-operator-5d7b7dc95c-kmnlf\" (UID: \"44087f53-56ca-43d7-aaf8-582ee3ba0c1c\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" Jul 01 19:23:47.468464 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.468425 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:47.495995 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.495912 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e53cdc86-b072-4c1e-9ca3-3d504c0bc6af-config\") pod \"console-operator-7fdbdf9c99-m82rf\" (UID: \"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:47.495995 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.495941 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-config-volume\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:47.495995 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.495961 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6frp\" (UniqueName: \"kubernetes.io/projected/5947ec3f-7e65-49cf-b511-44dc3b526ff2-kube-api-access-f6frp\") pod \"kube-storage-version-migrator-operator-79964ff6f9-685pf\" (UID: \"5947ec3f-7e65-49cf-b511-44dc3b526ff2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" Jul 01 19:23:47.495995 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.495985 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.496331 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496182 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/94519212-92c3-47b1-98d5-a1dfff195e48-registry-certificates\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.496331 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496226 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.496331 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496254 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-service-ca-bundle\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.496331 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:47.496298 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 01 19:23:47.496331 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:47.496318 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d9b4f4577-nvd9d: secret "image-registry-tls" not found Jul 01 19:23:47.496567 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496343 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/94519212-92c3-47b1-98d5-a1dfff195e48-installation-pull-secrets\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.496567 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:47.496382 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls podName:94519212-92c3-47b1-98d5-a1dfff195e48 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:47.996360889 +0000 UTC m=+35.542737510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls") pod "image-registry-6d9b4f4577-nvd9d" (UID: "94519212-92c3-47b1-98d5-a1dfff195e48") : secret "image-registry-tls" not found Jul 01 19:23:47.496567 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496428 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-rgndr\" (UID: \"9f78f69e-ffbb-4336-9976-daf5d2a9ea62\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" Jul 01 19:23:47.496567 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496463 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-snapshots\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.496567 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496490 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:47.496567 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496515 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-rgndr\" (UID: \"9f78f69e-ffbb-4336-9976-daf5d2a9ea62\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" Jul 01 19:23:47.496567 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496556 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-stats-auth\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:47.496907 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496583 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e53cdc86-b072-4c1e-9ca3-3d504c0bc6af-serving-cert\") pod \"console-operator-7fdbdf9c99-m82rf\" (UID: \"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:47.496907 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496620 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5947ec3f-7e65-49cf-b511-44dc3b526ff2-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-685pf\" (UID: \"5947ec3f-7e65-49cf-b511-44dc3b526ff2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" Jul 01 19:23:47.496907 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496646 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-config-volume\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:47.496907 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496707 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e53cdc86-b072-4c1e-9ca3-3d504c0bc6af-config\") pod \"console-operator-7fdbdf9c99-m82rf\" (UID: \"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:47.497143 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496906 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-service-ca-bundle\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.497143 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.496926 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/94519212-92c3-47b1-98d5-a1dfff195e48-registry-certificates\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.497244 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497136 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmlhr\" (UniqueName: \"kubernetes.io/projected/ef688413-3e26-4dfd-87c9-059133228994-kube-api-access-rmlhr\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:23:47.497244 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:47.497150 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle podName:0c94f978-1520-4f79-91df-f1f0d8048877 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:47.99713209 +0000 UTC m=+35.543508725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle") pod "router-default-6fb847847c-xqccn" (UID: "0c94f978-1520-4f79-91df-f1f0d8048877") : configmap references non-existent config key: service-ca.crt Jul 01 19:23:47.497349 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497312 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5947ec3f-7e65-49cf-b511-44dc3b526ff2-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-685pf\" (UID: \"5947ec3f-7e65-49cf-b511-44dc3b526ff2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" Jul 01 19:23:47.497400 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497354 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84j7l\" (UniqueName: \"kubernetes.io/projected/e53cdc86-b072-4c1e-9ca3-3d504c0bc6af-kube-api-access-84j7l\") pod \"console-operator-7fdbdf9c99-m82rf\" (UID: \"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:47.497400 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497378 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-tmp-dir\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:47.497501 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497411 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blswp\" (UniqueName: \"kubernetes.io/projected/5168c17b-70f2-4e40-ba13-d73744c17e7a-kube-api-access-blswp\") pod \"cluster-samples-operator-9ddc5ff76-2n74w\" (UID: \"5168c17b-70f2-4e40-ba13-d73744c17e7a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" Jul 01 19:23:47.497501 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497440 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94xpg\" (UniqueName: \"kubernetes.io/projected/d93cdf36-ab87-4ac4-b4a3-5ed00053c479-kube-api-access-94xpg\") pod \"volume-data-source-validator-69d9f679db-x7hw2\" (UID: \"d93cdf36-ab87-4ac4-b4a3-5ed00053c479\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-x7hw2" Jul 01 19:23:47.497501 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497459 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.497501 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497469 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-serving-cert\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.497501 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497499 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fjpl\" (UniqueName: \"kubernetes.io/projected/19decc52-62ea-4657-a5e7-dafa4b580eb8-kube-api-access-5fjpl\") pod \"network-check-source-56499946bc-sn7px\" (UID: \"19decc52-62ea-4657-a5e7-dafa4b580eb8\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-sn7px" Jul 01 19:23:47.497735 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497533 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-tmp\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.497735 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497560 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7hfj\" (UniqueName: \"kubernetes.io/projected/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-kube-api-access-b7hfj\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.497735 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497574 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-snapshots\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.497735 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497587 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/94519212-92c3-47b1-98d5-a1dfff195e48-image-registry-private-configuration\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.497735 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497622 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2jhr\" (UniqueName: \"kubernetes.io/projected/0c94f978-1520-4f79-91df-f1f0d8048877-kube-api-access-q2jhr\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:47.497735 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497645 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/952e2ea8-73dc-4456-9ae4-29fc1e2cdb02-cert\") pod \"ingress-canary-z82nd\" (UID: \"952e2ea8-73dc-4456-9ae4-29fc1e2cdb02\") " pod="openshift-ingress-canary/ingress-canary-z82nd" Jul 01 19:23:47.497735 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497687 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-default-certificate\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:47.497735 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497709 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:47.497735 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497735 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e53cdc86-b072-4c1e-9ca3-3d504c0bc6af-trusted-ca\") pod \"console-operator-7fdbdf9c99-m82rf\" (UID: \"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:47.498586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497758 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:47.498586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497781 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-bound-sa-token\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.498586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497805 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c57q8\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-kube-api-access-c57q8\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.498586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497831 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:23:47.498586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497833 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-tmp\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.498586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497856 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/94519212-92c3-47b1-98d5-a1dfff195e48-ca-trust-extracted\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.498586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497883 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-2n74w\" (UID: \"5168c17b-70f2-4e40-ba13-d73744c17e7a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" Jul 01 19:23:47.498586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497913 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ef688413-3e26-4dfd-87c9-059133228994-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:23:47.498586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497938 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-tmp-dir\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:47.498586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497953 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94519212-92c3-47b1-98d5-a1dfff195e48-trusted-ca\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.498586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.497979 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwph9\" (UniqueName: \"kubernetes.io/projected/952e2ea8-73dc-4456-9ae4-29fc1e2cdb02-kube-api-access-qwph9\") pod \"ingress-canary-z82nd\" (UID: \"952e2ea8-73dc-4456-9ae4-29fc1e2cdb02\") " pod="openshift-ingress-canary/ingress-canary-z82nd" Jul 01 19:23:47.498586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.498005 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9d8r\" (UniqueName: \"kubernetes.io/projected/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-kube-api-access-d9d8r\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:47.498586 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:47.498201 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 01 19:23:47.500876 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:47.498254 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs podName:0c94f978-1520-4f79-91df-f1f0d8048877 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:47.998236206 +0000 UTC m=+35.544612823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs") pod "router-default-6fb847847c-xqccn" (UID: "0c94f978-1520-4f79-91df-f1f0d8048877") : secret "router-metrics-certs-default" not found Jul 01 19:23:47.500876 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:47.498411 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 01 19:23:47.500876 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:47.498470 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls podName:ef688413-3e26-4dfd-87c9-059133228994 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:47.99845435 +0000 UTC m=+35.544830963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-gxrd5" (UID: "ef688413-3e26-4dfd-87c9-059133228994") : secret "cluster-monitoring-operator-tls" not found Jul 01 19:23:47.500876 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.498752 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/94519212-92c3-47b1-98d5-a1dfff195e48-ca-trust-extracted\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.500876 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:47.498833 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 01 19:23:47.500876 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:47.498874 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls podName:5168c17b-70f2-4e40-ba13-d73744c17e7a nodeName:}" failed. No retries permitted until 2026-07-01 19:23:47.998858886 +0000 UTC m=+35.545235497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-2n74w" (UID: "5168c17b-70f2-4e40-ba13-d73744c17e7a") : secret "samples-operator-tls" not found Jul 01 19:23:47.500876 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:47.499105 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 19:23:47.500876 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.499133 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e53cdc86-b072-4c1e-9ca3-3d504c0bc6af-trusted-ca\") pod \"console-operator-7fdbdf9c99-m82rf\" (UID: \"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:47.500876 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:47.499160 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls podName:9f285b32-ab0f-4888-bdc3-8c9f84a535cd nodeName:}" failed. No retries permitted until 2026-07-01 19:23:47.999144929 +0000 UTC m=+35.545521551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls") pod "dns-default-j8kx5" (UID: "9f285b32-ab0f-4888-bdc3-8c9f84a535cd") : secret "dns-default-metrics-tls" not found Jul 01 19:23:47.500876 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.499568 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ef688413-3e26-4dfd-87c9-059133228994-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:23:47.500876 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.499874 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-stats-auth\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:47.500876 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.499975 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94519212-92c3-47b1-98d5-a1dfff195e48-trusted-ca\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.501398 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.500116 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/94519212-92c3-47b1-98d5-a1dfff195e48-installation-pull-secrets\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.501398 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.500754 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/94519212-92c3-47b1-98d5-a1dfff195e48-image-registry-private-configuration\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.501398 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.500828 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-default-certificate\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:47.501398 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.501004 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e53cdc86-b072-4c1e-9ca3-3d504c0bc6af-serving-cert\") pod \"console-operator-7fdbdf9c99-m82rf\" (UID: \"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:47.501398 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.501357 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-serving-cert\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.508873 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.508845 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2jhr\" (UniqueName: \"kubernetes.io/projected/0c94f978-1520-4f79-91df-f1f0d8048877-kube-api-access-q2jhr\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:47.508990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.508880 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-bound-sa-token\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.509048 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.509003 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7hfj\" (UniqueName: \"kubernetes.io/projected/8f60a799-ece9-40f1-a3ff-44f5f10ef0a0-kube-api-access-b7hfj\") pod \"insights-operator-5f67b55555-428zv\" (UID: \"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0\") " pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.509859 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.509832 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blswp\" (UniqueName: \"kubernetes.io/projected/5168c17b-70f2-4e40-ba13-d73744c17e7a-kube-api-access-blswp\") pod \"cluster-samples-operator-9ddc5ff76-2n74w\" (UID: \"5168c17b-70f2-4e40-ba13-d73744c17e7a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" Jul 01 19:23:47.509958 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.509942 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94xpg\" (UniqueName: \"kubernetes.io/projected/d93cdf36-ab87-4ac4-b4a3-5ed00053c479-kube-api-access-94xpg\") pod \"volume-data-source-validator-69d9f679db-x7hw2\" (UID: \"d93cdf36-ab87-4ac4-b4a3-5ed00053c479\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-x7hw2" Jul 01 19:23:47.510011 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.509993 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c57q8\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-kube-api-access-c57q8\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:47.510851 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.510830 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmlhr\" (UniqueName: \"kubernetes.io/projected/ef688413-3e26-4dfd-87c9-059133228994-kube-api-access-rmlhr\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:23:47.511087 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.511053 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9d8r\" (UniqueName: \"kubernetes.io/projected/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-kube-api-access-d9d8r\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:47.511421 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.511407 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84j7l\" (UniqueName: \"kubernetes.io/projected/e53cdc86-b072-4c1e-9ca3-3d504c0bc6af-kube-api-access-84j7l\") pod \"console-operator-7fdbdf9c99-m82rf\" (UID: \"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:47.558557 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.558521 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" Jul 01 19:23:47.599112 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.599077 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwph9\" (UniqueName: \"kubernetes.io/projected/952e2ea8-73dc-4456-9ae4-29fc1e2cdb02-kube-api-access-qwph9\") pod \"ingress-canary-z82nd\" (UID: \"952e2ea8-73dc-4456-9ae4-29fc1e2cdb02\") " pod="openshift-ingress-canary/ingress-canary-z82nd" Jul 01 19:23:47.599112 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.599123 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6frp\" (UniqueName: \"kubernetes.io/projected/5947ec3f-7e65-49cf-b511-44dc3b526ff2-kube-api-access-f6frp\") pod \"kube-storage-version-migrator-operator-79964ff6f9-685pf\" (UID: \"5947ec3f-7e65-49cf-b511-44dc3b526ff2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" Jul 01 19:23:47.599345 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.599170 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-rgndr\" (UID: \"9f78f69e-ffbb-4336-9976-daf5d2a9ea62\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" Jul 01 19:23:47.599345 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.599203 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-rgndr\" (UID: \"9f78f69e-ffbb-4336-9976-daf5d2a9ea62\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" Jul 01 19:23:47.599345 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.599244 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5947ec3f-7e65-49cf-b511-44dc3b526ff2-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-685pf\" (UID: \"5947ec3f-7e65-49cf-b511-44dc3b526ff2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" Jul 01 19:23:47.599345 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.599275 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5947ec3f-7e65-49cf-b511-44dc3b526ff2-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-685pf\" (UID: \"5947ec3f-7e65-49cf-b511-44dc3b526ff2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" Jul 01 19:23:47.599345 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.599309 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fjpl\" (UniqueName: \"kubernetes.io/projected/19decc52-62ea-4657-a5e7-dafa4b580eb8-kube-api-access-5fjpl\") pod \"network-check-source-56499946bc-sn7px\" (UID: \"19decc52-62ea-4657-a5e7-dafa4b580eb8\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-sn7px" Jul 01 19:23:47.599345 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.599346 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/952e2ea8-73dc-4456-9ae4-29fc1e2cdb02-cert\") pod \"ingress-canary-z82nd\" (UID: \"952e2ea8-73dc-4456-9ae4-29fc1e2cdb02\") " pod="openshift-ingress-canary/ingress-canary-z82nd" Jul 01 19:23:47.599741 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:47.599716 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 01 19:23:47.599814 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:47.599795 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert podName:9f78f69e-ffbb-4336-9976-daf5d2a9ea62 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:48.099771612 +0000 UTC m=+35.646148235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-rgndr" (UID: "9f78f69e-ffbb-4336-9976-daf5d2a9ea62") : secret "networking-console-plugin-cert" not found Jul 01 19:23:47.599903 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.599876 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5947ec3f-7e65-49cf-b511-44dc3b526ff2-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-685pf\" (UID: \"5947ec3f-7e65-49cf-b511-44dc3b526ff2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" Jul 01 19:23:47.600028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.599976 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-rgndr\" (UID: \"9f78f69e-ffbb-4336-9976-daf5d2a9ea62\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" Jul 01 19:23:47.601701 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.601684 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5947ec3f-7e65-49cf-b511-44dc3b526ff2-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-685pf\" (UID: \"5947ec3f-7e65-49cf-b511-44dc3b526ff2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" Jul 01 19:23:47.601973 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.601954 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/952e2ea8-73dc-4456-9ae4-29fc1e2cdb02-cert\") pod \"ingress-canary-z82nd\" (UID: \"952e2ea8-73dc-4456-9ae4-29fc1e2cdb02\") " pod="openshift-ingress-canary/ingress-canary-z82nd" Jul 01 19:23:47.611959 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.611930 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6frp\" (UniqueName: \"kubernetes.io/projected/5947ec3f-7e65-49cf-b511-44dc3b526ff2-kube-api-access-f6frp\") pod \"kube-storage-version-migrator-operator-79964ff6f9-685pf\" (UID: \"5947ec3f-7e65-49cf-b511-44dc3b526ff2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" Jul 01 19:23:47.612105 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.612007 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwph9\" (UniqueName: \"kubernetes.io/projected/952e2ea8-73dc-4456-9ae4-29fc1e2cdb02-kube-api-access-qwph9\") pod \"ingress-canary-z82nd\" (UID: \"952e2ea8-73dc-4456-9ae4-29fc1e2cdb02\") " pod="openshift-ingress-canary/ingress-canary-z82nd" Jul 01 19:23:47.613298 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.613274 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fjpl\" (UniqueName: \"kubernetes.io/projected/19decc52-62ea-4657-a5e7-dafa4b580eb8-kube-api-access-5fjpl\") pod \"network-check-source-56499946bc-sn7px\" (UID: \"19decc52-62ea-4657-a5e7-dafa4b580eb8\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-sn7px" Jul 01 19:23:47.658004 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.657971 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-428zv" Jul 01 19:23:47.663125 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.663046 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-x7hw2" Jul 01 19:23:47.674632 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.674580 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:47.692189 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.692160 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-sn7px" Jul 01 19:23:47.698313 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.698290 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" Jul 01 19:23:47.708313 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.708288 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z82nd" Jul 01 19:23:47.718469 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.718252 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf"] Jul 01 19:23:47.724015 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:47.723971 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44087f53_56ca_43d7_aaf8_582ee3ba0c1c.slice/crio-2a8d769f1fd09816d742ba39495ff3bf9f5a79416d10d5f8797ad2d6ceeb3174 WatchSource:0}: Error finding container 2a8d769f1fd09816d742ba39495ff3bf9f5a79416d10d5f8797ad2d6ceeb3174: Status 404 returned error can't find the container with id 2a8d769f1fd09816d742ba39495ff3bf9f5a79416d10d5f8797ad2d6ceeb3174 Jul 01 19:23:47.756320 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.756234 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:47.825478 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.823415 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:47.860250 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.860174 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-428zv"] Jul 01 19:23:47.874769 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:47.874709 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f60a799_ece9_40f1_a3ff_44f5f10ef0a0.slice/crio-5f124609cb2cfaa0e069747331f54dd08c4984ce20d2f3fd437c8a6339eff445 WatchSource:0}: Error finding container 5f124609cb2cfaa0e069747331f54dd08c4984ce20d2f3fd437c8a6339eff445: Status 404 returned error can't find the container with id 5f124609cb2cfaa0e069747331f54dd08c4984ce20d2f3fd437c8a6339eff445 Jul 01 19:23:47.882671 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.882308 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-x7hw2"] Jul 01 19:23:47.893440 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:47.893309 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93cdf36_ab87_4ac4_b4a3_5ed00053c479.slice/crio-54d876b8b65bea16a5be81c321d0a28bd4fb5c67b139c68cd483962c12df5669 WatchSource:0}: Error finding container 54d876b8b65bea16a5be81c321d0a28bd4fb5c67b139c68cd483962c12df5669: Status 404 returned error can't find the container with id 54d876b8b65bea16a5be81c321d0a28bd4fb5c67b139c68cd483962c12df5669 Jul 01 19:23:47.927414 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.927385 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf"] Jul 01 19:23:47.930038 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.930003 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-sn7px"] Jul 01 19:23:47.932019 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:47.931938 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5947ec3f_7e65_49cf_b511_44dc3b526ff2.slice/crio-e7cb050353915abca04b0b02727b16b62b146673a9ec6c4c90ab9d78d9696460 WatchSource:0}: Error finding container e7cb050353915abca04b0b02727b16b62b146673a9ec6c4c90ab9d78d9696460: Status 404 returned error can't find the container with id e7cb050353915abca04b0b02727b16b62b146673a9ec6c4c90ab9d78d9696460 Jul 01 19:23:47.935868 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:47.935845 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19decc52_62ea_4657_a5e7_dafa4b580eb8.slice/crio-53c4b5b06b348dcc629b651f8ea6e75fbdd33cd3989ff61f40e6b262b3007381 WatchSource:0}: Error finding container 53c4b5b06b348dcc629b651f8ea6e75fbdd33cd3989ff61f40e6b262b3007381: Status 404 returned error can't find the container with id 53c4b5b06b348dcc629b651f8ea6e75fbdd33cd3989ff61f40e6b262b3007381 Jul 01 19:23:47.955779 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.955601 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:23:47.956891 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.956865 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-m82rf"] Jul 01 19:23:47.958047 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.958032 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-64bmf\"" Jul 01 19:23:47.959442 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:47.959422 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53cdc86_b072_4c1e_9ca3_3d504c0bc6af.slice/crio-e2f6caa9d0baed4ba692b918cc50ed1fd8d8f349ed044d7a28a96c5683ee6abf WatchSource:0}: Error finding container e2f6caa9d0baed4ba692b918cc50ed1fd8d8f349ed044d7a28a96c5683ee6abf: Status 404 returned error can't find the container with id e2f6caa9d0baed4ba692b918cc50ed1fd8d8f349ed044d7a28a96c5683ee6abf Jul 01 19:23:47.965782 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:47.965763 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z82nd"] Jul 01 19:23:47.967644 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:47.967620 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952e2ea8_73dc_4456_9ae4_29fc1e2cdb02.slice/crio-fff2968cf66071788a1552aa3c9da73bb731af83e770b6088f00fb1feebc1132 WatchSource:0}: Error finding container fff2968cf66071788a1552aa3c9da73bb731af83e770b6088f00fb1feebc1132: Status 404 returned error can't find the container with id fff2968cf66071788a1552aa3c9da73bb731af83e770b6088f00fb1feebc1132 Jul 01 19:23:48.003214 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.003154 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:48.003214 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.003203 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:48.003371 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.003298 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:48.003371 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:48.003306 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 01 19:23:48.003371 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:48.003322 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d9b4f4577-nvd9d: secret "image-registry-tls" not found Jul 01 19:23:48.003371 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.003329 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:48.003371 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.003357 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:23:48.003371 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:48.003371 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls podName:94519212-92c3-47b1-98d5-a1dfff195e48 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:49.003354234 +0000 UTC m=+36.549730860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls") pod "image-registry-6d9b4f4577-nvd9d" (UID: "94519212-92c3-47b1-98d5-a1dfff195e48") : secret "image-registry-tls" not found Jul 01 19:23:48.003570 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:48.003391 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 01 19:23:48.003570 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:48.003408 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 19:23:48.003570 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:48.003421 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 01 19:23:48.003570 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:48.003442 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle podName:0c94f978-1520-4f79-91df-f1f0d8048877 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:49.003423748 +0000 UTC m=+36.549800371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle") pod "router-default-6fb847847c-xqccn" (UID: "0c94f978-1520-4f79-91df-f1f0d8048877") : configmap references non-existent config key: service-ca.crt Jul 01 19:23:48.003570 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:48.003463 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs podName:0c94f978-1520-4f79-91df-f1f0d8048877 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:49.003456208 +0000 UTC m=+36.549832824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs") pod "router-default-6fb847847c-xqccn" (UID: "0c94f978-1520-4f79-91df-f1f0d8048877") : secret "router-metrics-certs-default" not found Jul 01 19:23:48.003570 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:48.003475 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls podName:9f285b32-ab0f-4888-bdc3-8c9f84a535cd nodeName:}" failed. No retries permitted until 2026-07-01 19:23:49.003469799 +0000 UTC m=+36.549846411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls") pod "dns-default-j8kx5" (UID: "9f285b32-ab0f-4888-bdc3-8c9f84a535cd") : secret "dns-default-metrics-tls" not found Jul 01 19:23:48.003570 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:48.003495 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls podName:ef688413-3e26-4dfd-87c9-059133228994 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:49.003490272 +0000 UTC m=+36.549866884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-gxrd5" (UID: "ef688413-3e26-4dfd-87c9-059133228994") : secret "cluster-monitoring-operator-tls" not found Jul 01 19:23:48.003570 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.003516 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-2n74w\" (UID: \"5168c17b-70f2-4e40-ba13-d73744c17e7a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" Jul 01 19:23:48.003837 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:48.003598 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 01 19:23:48.003837 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:48.003622 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls podName:5168c17b-70f2-4e40-ba13-d73744c17e7a nodeName:}" failed. No retries permitted until 2026-07-01 19:23:49.00361627 +0000 UTC m=+36.549992882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-2n74w" (UID: "5168c17b-70f2-4e40-ba13-d73744c17e7a") : secret "samples-operator-tls" not found Jul 01 19:23:48.102929 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.102894 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z82nd" event={"ID":"952e2ea8-73dc-4456-9ae4-29fc1e2cdb02","Type":"ContainerStarted","Data":"fff2968cf66071788a1552aa3c9da73bb731af83e770b6088f00fb1feebc1132"} Jul 01 19:23:48.103903 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.103873 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-428zv" event={"ID":"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0","Type":"ContainerStarted","Data":"5f124609cb2cfaa0e069747331f54dd08c4984ce20d2f3fd437c8a6339eff445"} Jul 01 19:23:48.104018 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.103945 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-rgndr\" (UID: \"9f78f69e-ffbb-4336-9976-daf5d2a9ea62\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" Jul 01 19:23:48.104146 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:48.104091 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 01 19:23:48.104198 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:48.104157 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert podName:9f78f69e-ffbb-4336-9976-daf5d2a9ea62 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:49.104136758 +0000 UTC m=+36.650513369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-rgndr" (UID: "9f78f69e-ffbb-4336-9976-daf5d2a9ea62") : secret "networking-console-plugin-cert" not found Jul 01 19:23:48.104978 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.104955 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" event={"ID":"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af","Type":"ContainerStarted","Data":"e2f6caa9d0baed4ba692b918cc50ed1fd8d8f349ed044d7a28a96c5683ee6abf"} Jul 01 19:23:48.106016 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.105997 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-x7hw2" event={"ID":"d93cdf36-ab87-4ac4-b4a3-5ed00053c479","Type":"ContainerStarted","Data":"54d876b8b65bea16a5be81c321d0a28bd4fb5c67b139c68cd483962c12df5669"} Jul 01 19:23:48.107081 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.107048 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" event={"ID":"44087f53-56ca-43d7-aaf8-582ee3ba0c1c","Type":"ContainerStarted","Data":"2a8d769f1fd09816d742ba39495ff3bf9f5a79416d10d5f8797ad2d6ceeb3174"} Jul 01 19:23:48.109798 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.109777 2585 generic.go:356] "Generic (PLEG): container finished" podID="90fc2dc7-c463-49e5-b2f5-050c9238f37c" containerID="2b82299e1bfabc6cf485b1ff8c9eb8f85df26a9569f5ed5fe1d80cafd9fcc007" exitCode=0 Jul 01 19:23:48.109891 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.109847 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plcl5" event={"ID":"90fc2dc7-c463-49e5-b2f5-050c9238f37c","Type":"ContainerDied","Data":"2b82299e1bfabc6cf485b1ff8c9eb8f85df26a9569f5ed5fe1d80cafd9fcc007"} Jul 01 19:23:48.110849 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.110824 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" event={"ID":"5947ec3f-7e65-49cf-b511-44dc3b526ff2","Type":"ContainerStarted","Data":"e7cb050353915abca04b0b02727b16b62b146673a9ec6c4c90ab9d78d9696460"} Jul 01 19:23:48.111823 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.111805 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-sn7px" event={"ID":"19decc52-62ea-4657-a5e7-dafa4b580eb8","Type":"ContainerStarted","Data":"53c4b5b06b348dcc629b651f8ea6e75fbdd33cd3989ff61f40e6b262b3007381"} Jul 01 19:23:48.283664 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.283612 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:48.366443 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.366411 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:48.498934 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.498872 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:48.770190 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.770105 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:48.959983 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.959200 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:23:48.964336 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.960656 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:23:48.964336 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.961908 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4nv7g\"" Jul 01 19:23:48.964336 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.962147 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 01 19:23:48.966039 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:48.964815 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 01 19:23:49.014945 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:49.014911 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:49.015142 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:49.014961 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:49.015142 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:49.014993 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:23:49.015142 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:49.015023 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-2n74w\" (UID: \"5168c17b-70f2-4e40-ba13-d73744c17e7a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" Jul 01 19:23:49.015142 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:49.015085 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:49.015371 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:49.015168 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:49.015422 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:49.015406 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle podName:0c94f978-1520-4f79-91df-f1f0d8048877 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:51.015387846 +0000 UTC m=+38.561764479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle") pod "router-default-6fb847847c-xqccn" (UID: "0c94f978-1520-4f79-91df-f1f0d8048877") : configmap references non-existent config key: service-ca.crt Jul 01 19:23:49.016835 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:49.015777 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 01 19:23:49.016835 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:49.015810 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 01 19:23:49.016835 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:49.015825 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d9b4f4577-nvd9d: secret "image-registry-tls" not found Jul 01 19:23:49.016835 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:49.015856 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls podName:5168c17b-70f2-4e40-ba13-d73744c17e7a nodeName:}" failed. No retries permitted until 2026-07-01 19:23:51.015839062 +0000 UTC m=+38.562215689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-2n74w" (UID: "5168c17b-70f2-4e40-ba13-d73744c17e7a") : secret "samples-operator-tls" not found Jul 01 19:23:49.016835 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:49.015877 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls podName:94519212-92c3-47b1-98d5-a1dfff195e48 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:51.015866657 +0000 UTC m=+38.562243274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls") pod "image-registry-6d9b4f4577-nvd9d" (UID: "94519212-92c3-47b1-98d5-a1dfff195e48") : secret "image-registry-tls" not found Jul 01 19:23:49.016835 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:49.015932 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 01 19:23:49.016835 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:49.015963 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs podName:0c94f978-1520-4f79-91df-f1f0d8048877 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:51.015953198 +0000 UTC m=+38.562329832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs") pod "router-default-6fb847847c-xqccn" (UID: "0c94f978-1520-4f79-91df-f1f0d8048877") : secret "router-metrics-certs-default" not found Jul 01 19:23:49.016835 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:49.016010 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 19:23:49.016835 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:49.016038 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls podName:9f285b32-ab0f-4888-bdc3-8c9f84a535cd nodeName:}" failed. No retries permitted until 2026-07-01 19:23:51.016028402 +0000 UTC m=+38.562405027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls") pod "dns-default-j8kx5" (UID: "9f285b32-ab0f-4888-bdc3-8c9f84a535cd") : secret "dns-default-metrics-tls" not found Jul 01 19:23:49.016835 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:49.016103 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 01 19:23:49.016835 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:49.016134 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls podName:ef688413-3e26-4dfd-87c9-059133228994 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:51.016124737 +0000 UTC m=+38.562501363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-gxrd5" (UID: "ef688413-3e26-4dfd-87c9-059133228994") : secret "cluster-monitoring-operator-tls" not found Jul 01 19:23:49.035708 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:49.035659 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:49.115999 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:49.115961 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-rgndr\" (UID: \"9f78f69e-ffbb-4336-9976-daf5d2a9ea62\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" Jul 01 19:23:49.116206 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:49.116138 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 01 19:23:49.116206 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:49.116197 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert podName:9f78f69e-ffbb-4336-9976-daf5d2a9ea62 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:51.116182665 +0000 UTC m=+38.662559280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-rgndr" (UID: "9f78f69e-ffbb-4336-9976-daf5d2a9ea62") : secret "networking-console-plugin-cert" not found Jul 01 19:23:49.121433 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:49.121310 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plcl5" event={"ID":"90fc2dc7-c463-49e5-b2f5-050c9238f37c","Type":"ContainerStarted","Data":"1742dd7658acbac3efe4abf2251c5cdbc48ecc706ca2d7c2d0fa45afaf9326ee"} Jul 01 19:23:49.148147 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:49.148087 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-plcl5" podStartSLOduration=7.801929165 podStartE2EDuration="36.148049299s" totalImagesPullingTime="28.346120134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:23:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:49.147267359 +0000 UTC m=+36.693644191" watchObservedRunningTime="2026-07-01 19:23:49.148049299 +0000 UTC m=+36.694425937" Jul 01 19:23:49.271600 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:49.271557 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:49.370247 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:49.370093 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:51.038620 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:51.038525 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:51.038620 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:51.038583 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:23:51.038620 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:51.038614 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-2n74w\" (UID: \"5168c17b-70f2-4e40-ba13-d73744c17e7a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" Jul 01 19:23:51.039239 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:51.038664 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:51.039239 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:51.038706 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 19:23:51.039239 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:51.038716 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:51.039239 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:51.038761 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 01 19:23:51.039239 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:51.038825 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls podName:9f285b32-ab0f-4888-bdc3-8c9f84a535cd nodeName:}" failed. No retries permitted until 2026-07-01 19:23:55.03880613 +0000 UTC m=+42.585182744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls") pod "dns-default-j8kx5" (UID: "9f285b32-ab0f-4888-bdc3-8c9f84a535cd") : secret "dns-default-metrics-tls" not found Jul 01 19:23:51.039239 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:51.038889 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 01 19:23:51.039239 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:51.038902 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d9b4f4577-nvd9d: secret "image-registry-tls" not found Jul 01 19:23:51.039239 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:51.038952 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls podName:ef688413-3e26-4dfd-87c9-059133228994 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:55.03893798 +0000 UTC m=+42.585314591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-gxrd5" (UID: "ef688413-3e26-4dfd-87c9-059133228994") : secret "cluster-monitoring-operator-tls" not found Jul 01 19:23:51.039239 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:51.038972 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle podName:0c94f978-1520-4f79-91df-f1f0d8048877 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:55.038962991 +0000 UTC m=+42.585339604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle") pod "router-default-6fb847847c-xqccn" (UID: "0c94f978-1520-4f79-91df-f1f0d8048877") : configmap references non-existent config key: service-ca.crt Jul 01 19:23:51.039239 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:51.038978 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 01 19:23:51.039239 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:51.038995 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls podName:94519212-92c3-47b1-98d5-a1dfff195e48 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:55.038987258 +0000 UTC m=+42.585363871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls") pod "image-registry-6d9b4f4577-nvd9d" (UID: "94519212-92c3-47b1-98d5-a1dfff195e48") : secret "image-registry-tls" not found Jul 01 19:23:51.039239 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:51.039010 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls podName:5168c17b-70f2-4e40-ba13-d73744c17e7a nodeName:}" failed. No retries permitted until 2026-07-01 19:23:55.039004002 +0000 UTC m=+42.585380619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-2n74w" (UID: "5168c17b-70f2-4e40-ba13-d73744c17e7a") : secret "samples-operator-tls" not found Jul 01 19:23:51.039239 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:51.039109 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:51.039757 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:51.039311 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 01 19:23:51.039757 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:51.039356 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs podName:0c94f978-1520-4f79-91df-f1f0d8048877 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:55.039342818 +0000 UTC m=+42.585719444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs") pod "router-default-6fb847847c-xqccn" (UID: "0c94f978-1520-4f79-91df-f1f0d8048877") : secret "router-metrics-certs-default" not found Jul 01 19:23:51.140194 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:51.140110 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-rgndr\" (UID: \"9f78f69e-ffbb-4336-9976-daf5d2a9ea62\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" Jul 01 19:23:51.140329 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:51.140259 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 01 19:23:51.140329 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:51.140321 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert podName:9f78f69e-ffbb-4336-9976-daf5d2a9ea62 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:55.140302064 +0000 UTC m=+42.686678693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-rgndr" (UID: "9f78f69e-ffbb-4336-9976-daf5d2a9ea62") : secret "networking-console-plugin-cert" not found Jul 01 19:23:55.074411 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:55.074370 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:55.074866 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:55.074489 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:23:55.074866 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:55.074519 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:23:55.074866 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:55.074540 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:23:55.074866 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:55.074571 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle podName:0c94f978-1520-4f79-91df-f1f0d8048877 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:03.074547409 +0000 UTC m=+50.620924026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle") pod "router-default-6fb847847c-xqccn" (UID: "0c94f978-1520-4f79-91df-f1f0d8048877") : configmap references non-existent config key: service-ca.crt Jul 01 19:23:55.074866 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:55.074609 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 01 19:23:55.074866 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:55.074625 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-2n74w\" (UID: \"5168c17b-70f2-4e40-ba13-d73744c17e7a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" Jul 01 19:23:55.074866 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:55.074657 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls podName:ef688413-3e26-4dfd-87c9-059133228994 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:03.074642392 +0000 UTC m=+50.621019020 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-gxrd5" (UID: "ef688413-3e26-4dfd-87c9-059133228994") : secret "cluster-monitoring-operator-tls" not found Jul 01 19:23:55.074866 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:55.074579 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 01 19:23:55.074866 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:55.074668 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 19:23:55.074866 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:55.074697 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 01 19:23:55.074866 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:55.074708 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs podName:0c94f978-1520-4f79-91df-f1f0d8048877 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:03.07469268 +0000 UTC m=+50.621069292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs") pod "router-default-6fb847847c-xqccn" (UID: "0c94f978-1520-4f79-91df-f1f0d8048877") : secret "router-metrics-certs-default" not found Jul 01 19:23:55.074866 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:55.074733 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls podName:5168c17b-70f2-4e40-ba13-d73744c17e7a nodeName:}" failed. No retries permitted until 2026-07-01 19:24:03.074722196 +0000 UTC m=+50.621098815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-2n74w" (UID: "5168c17b-70f2-4e40-ba13-d73744c17e7a") : secret "samples-operator-tls" not found Jul 01 19:23:55.074866 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:55.074732 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:23:55.075373 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:55.074747 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls podName:9f285b32-ab0f-4888-bdc3-8c9f84a535cd nodeName:}" failed. No retries permitted until 2026-07-01 19:24:03.074739412 +0000 UTC m=+50.621116031 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls") pod "dns-default-j8kx5" (UID: "9f285b32-ab0f-4888-bdc3-8c9f84a535cd") : secret "dns-default-metrics-tls" not found Jul 01 19:23:55.075373 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:55.074799 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 01 19:23:55.075373 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:55.074810 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d9b4f4577-nvd9d: secret "image-registry-tls" not found Jul 01 19:23:55.075373 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:55.074842 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls podName:94519212-92c3-47b1-98d5-a1dfff195e48 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:03.074831889 +0000 UTC m=+50.621208518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls") pod "image-registry-6d9b4f4577-nvd9d" (UID: "94519212-92c3-47b1-98d5-a1dfff195e48") : secret "image-registry-tls" not found Jul 01 19:23:55.176081 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:55.176036 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-rgndr\" (UID: \"9f78f69e-ffbb-4336-9976-daf5d2a9ea62\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" Jul 01 19:23:55.176242 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:55.176212 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 01 19:23:55.176314 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:55.176302 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert podName:9f78f69e-ffbb-4336-9976-daf5d2a9ea62 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:03.176281468 +0000 UTC m=+50.722658101 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-rgndr" (UID: "9f78f69e-ffbb-4336-9976-daf5d2a9ea62") : secret "networking-console-plugin-cert" not found Jul 01 19:23:55.639286 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:55.639168 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:56.142672 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.142642 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/0.log" Jul 01 19:23:56.143152 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.142691 2585 generic.go:356] "Generic (PLEG): container finished" podID="e53cdc86-b072-4c1e-9ca3-3d504c0bc6af" containerID="0e2205a75e960b5aa46a0e51ac2149767ad92a9d3ce357fc16c5a4f3b542614f" exitCode=255 Jul 01 19:23:56.143152 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.142772 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" event={"ID":"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af","Type":"ContainerDied","Data":"0e2205a75e960b5aa46a0e51ac2149767ad92a9d3ce357fc16c5a4f3b542614f"} Jul 01 19:23:56.143152 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.143052 2585 scope.go:117] "RemoveContainer" containerID="0e2205a75e960b5aa46a0e51ac2149767ad92a9d3ce357fc16c5a4f3b542614f" Jul 01 19:23:56.144830 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.144803 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-x7hw2" event={"ID":"d93cdf36-ab87-4ac4-b4a3-5ed00053c479","Type":"ContainerStarted","Data":"bd1ce1b9ab803aef51e1deefe01972a829bf7de8ae9c9d92598b7fc267c07e4c"} Jul 01 19:23:56.146411 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.146387 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" event={"ID":"44087f53-56ca-43d7-aaf8-582ee3ba0c1c","Type":"ContainerStarted","Data":"d8d81543cb522657137bd5b42b1782050d2fe7d0ed312f4d815d28e1732bcca2"} Jul 01 19:23:56.148014 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.147988 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" event={"ID":"5947ec3f-7e65-49cf-b511-44dc3b526ff2","Type":"ContainerStarted","Data":"03b35db84350a38e6d4639d38f8b47bfda13527ce084ed9435942dbe809034f2"} Jul 01 19:23:56.149878 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.149856 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-sn7px" event={"ID":"19decc52-62ea-4657-a5e7-dafa4b580eb8","Type":"ContainerStarted","Data":"6f84e3574da998b785f9a9f44e69a4d05f2e7634fb69e1b31008756d7bf32477"} Jul 01 19:23:56.152457 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.152437 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z82nd" event={"ID":"952e2ea8-73dc-4456-9ae4-29fc1e2cdb02","Type":"ContainerStarted","Data":"0d6a2e465e9a558e2b5d96310596b1b156138e1c0fad4d137989aecd28816b20"} Jul 01 19:23:56.154757 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.154736 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-428zv" event={"ID":"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0","Type":"ContainerStarted","Data":"8c3dd23869130a38239be0f50fdb7aff8c0717eb66f145d634ce875a37a10346"} Jul 01 19:23:56.176899 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.176859 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-x7hw2" podStartSLOduration=19.483548339 podStartE2EDuration="26.176846093s" totalImagesPullingTime="6.693297754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:56.176747299 +0000 UTC m=+43.723123934" watchObservedRunningTime="2026-07-01 19:23:56.176846093 +0000 UTC m=+43.723222720" Jul 01 19:23:56.196585 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.196553 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5f67b55555-428zv" podStartSLOduration=18.586127406 podStartE2EDuration="26.196538427s" totalImagesPullingTime="7.610411021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:56.195213155 +0000 UTC m=+43.741589790" watchObservedRunningTime="2026-07-01 19:23:56.196538427 +0000 UTC m=+43.742915040" Jul 01 19:23:56.217622 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.217580 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-56499946bc-sn7px" podStartSLOduration=18.698829773 podStartE2EDuration="26.217566287s" totalImagesPullingTime="7.518736514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:56.217035324 +0000 UTC m=+43.763411951" watchObservedRunningTime="2026-07-01 19:23:56.217566287 +0000 UTC m=+43.763942920" Jul 01 19:23:56.239871 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.238820 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" podStartSLOduration=18.716687169 podStartE2EDuration="26.238802764s" totalImagesPullingTime="7.522115595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:56.237992714 +0000 UTC m=+43.784369348" watchObservedRunningTime="2026-07-01 19:23:56.238802764 +0000 UTC m=+43.785179399" Jul 01 19:23:56.277211 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.277147 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:56.289386 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.285860 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" podStartSLOduration=18.555446147 podStartE2EDuration="26.28584015s" totalImagesPullingTime="7.730394003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:56.285407279 +0000 UTC m=+43.831783914" watchObservedRunningTime="2026-07-01 19:23:56.28584015 +0000 UTC m=+43.832216786" Jul 01 19:23:56.289386 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.286321 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z82nd" podStartSLOduration=1.7754644590000002 podStartE2EDuration="9.286295158s" totalImagesPullingTime="7.510830699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:56.257677224 +0000 UTC m=+43.804053858" watchObservedRunningTime="2026-07-01 19:23:56.286295158 +0000 UTC m=+43.832671795" Jul 01 19:23:56.377594 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.377552 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:56.787857 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.787791 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:56.812391 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.812346 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-nnfgg"] Jul 01 19:23:56.833600 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.833569 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-nnfgg"] Jul 01 19:23:56.833785 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.833693 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nnfgg" Jul 01 19:23:56.836318 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.836278 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 01 19:23:56.836442 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.836323 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zdb9v\"" Jul 01 19:23:56.836442 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.836289 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 01 19:23:56.860701 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.860674 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:56.938566 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.938529 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:56.994190 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:56.994159 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf2pc\" (UniqueName: \"kubernetes.io/projected/9070bac0-2f2a-4fc2-b184-7363d8aa12ac-kube-api-access-wf2pc\") pod \"migrator-786f9fbc5-nnfgg\" (UID: \"9070bac0-2f2a-4fc2-b184-7363d8aa12ac\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nnfgg" Jul 01 19:23:57.095802 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.095715 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf2pc\" (UniqueName: \"kubernetes.io/projected/9070bac0-2f2a-4fc2-b184-7363d8aa12ac-kube-api-access-wf2pc\") pod \"migrator-786f9fbc5-nnfgg\" (UID: \"9070bac0-2f2a-4fc2-b184-7363d8aa12ac\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nnfgg" Jul 01 19:23:57.105141 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.105117 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf2pc\" (UniqueName: \"kubernetes.io/projected/9070bac0-2f2a-4fc2-b184-7363d8aa12ac-kube-api-access-wf2pc\") pod \"migrator-786f9fbc5-nnfgg\" (UID: \"9070bac0-2f2a-4fc2-b184-7363d8aa12ac\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nnfgg" Jul 01 19:23:57.145471 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.145436 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nnfgg" Jul 01 19:23:57.159466 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.159439 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/1.log" Jul 01 19:23:57.159912 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.159896 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/0.log" Jul 01 19:23:57.159994 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.159937 2585 generic.go:356] "Generic (PLEG): container finished" podID="e53cdc86-b072-4c1e-9ca3-3d504c0bc6af" containerID="197b66fc4a7bee20a6091345d94a15f2bfcef1355798b9af5b7773669388e004" exitCode=255 Jul 01 19:23:57.160044 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.159981 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" event={"ID":"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af","Type":"ContainerDied","Data":"197b66fc4a7bee20a6091345d94a15f2bfcef1355798b9af5b7773669388e004"} Jul 01 19:23:57.160127 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.160087 2585 scope.go:117] "RemoveContainer" containerID="0e2205a75e960b5aa46a0e51ac2149767ad92a9d3ce357fc16c5a4f3b542614f" Jul 01 19:23:57.160373 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.160352 2585 scope.go:117] "RemoveContainer" containerID="197b66fc4a7bee20a6091345d94a15f2bfcef1355798b9af5b7773669388e004" Jul 01 19:23:57.160630 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:57.160603 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-m82rf_openshift-console-operator(e53cdc86-b072-4c1e-9ca3-3d504c0bc6af)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" podUID="e53cdc86-b072-4c1e-9ca3-3d504c0bc6af" Jul 01 19:23:57.185219 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.185187 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:57.248571 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.248529 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:57.298547 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.298356 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-nnfgg"] Jul 01 19:23:57.300450 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:23:57.300427 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9070bac0_2f2a_4fc2_b184_7363d8aa12ac.slice/crio-18af6cd36cfae179253c0a77a390a9fab8cacc967419d109c352d65a29df49ae WatchSource:0}: Error finding container 18af6cd36cfae179253c0a77a390a9fab8cacc967419d109c352d65a29df49ae: Status 404 returned error can't find the container with id 18af6cd36cfae179253c0a77a390a9fab8cacc967419d109c352d65a29df49ae Jul 01 19:23:57.434220 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.434128 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:57.533335 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.533293 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:57.675269 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.675230 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:57.675433 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.675285 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:23:57.896981 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:57.896947 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:58.002880 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.002826 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:58.035755 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.035719 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tdm9z"] Jul 01 19:23:58.050523 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.050486 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.053422 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.053398 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 01 19:23:58.054728 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.054699 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 01 19:23:58.054835 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.054712 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5hd6c\"" Jul 01 19:23:58.062355 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.062231 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tdm9z"] Jul 01 19:23:58.092131 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.092099 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:58.164617 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.164547 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/1.log" Jul 01 19:23:58.165115 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.164903 2585 scope.go:117] "RemoveContainer" containerID="197b66fc4a7bee20a6091345d94a15f2bfcef1355798b9af5b7773669388e004" Jul 01 19:23:58.165200 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:58.165121 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-m82rf_openshift-console-operator(e53cdc86-b072-4c1e-9ca3-3d504c0bc6af)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" podUID="e53cdc86-b072-4c1e-9ca3-3d504c0bc6af" Jul 01 19:23:58.165913 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.165891 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nnfgg" event={"ID":"9070bac0-2f2a-4fc2-b184-7363d8aa12ac","Type":"ContainerStarted","Data":"18af6cd36cfae179253c0a77a390a9fab8cacc967419d109c352d65a29df49ae"} Jul 01 19:23:58.173034 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.173007 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:58.205523 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.205489 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0510fe39-50bf-472c-adc1-f078f5406fdd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.205663 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.205570 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.205663 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.205623 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0510fe39-50bf-472c-adc1-f078f5406fdd-crio-socket\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.205663 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.205651 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0510fe39-50bf-472c-adc1-f078f5406fdd-data-volume\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.205771 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.205715 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5fd5\" (UniqueName: \"kubernetes.io/projected/0510fe39-50bf-472c-adc1-f078f5406fdd-kube-api-access-h5fd5\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.306420 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.306380 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0510fe39-50bf-472c-adc1-f078f5406fdd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.306583 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.306461 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.306583 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:58.306509 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 01 19:23:58.306583 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.306517 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0510fe39-50bf-472c-adc1-f078f5406fdd-crio-socket\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.306733 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:58.306580 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0510fe39-50bf-472c-adc1-f078f5406fdd-kube-rbac-proxy-cm podName:0510fe39-50bf-472c-adc1-f078f5406fdd nodeName:}" failed. No retries permitted until 2026-07-01 19:23:58.806561638 +0000 UTC m=+46.352938255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0510fe39-50bf-472c-adc1-f078f5406fdd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdm9z" (UID: "0510fe39-50bf-472c-adc1-f078f5406fdd") : configmap "kube-rbac-proxy" not found Jul 01 19:23:58.306733 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:58.306617 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 01 19:23:58.306733 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.306648 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0510fe39-50bf-472c-adc1-f078f5406fdd-data-volume\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.306733 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:58.306665 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls podName:0510fe39-50bf-472c-adc1-f078f5406fdd nodeName:}" failed. No retries permitted until 2026-07-01 19:23:58.806650555 +0000 UTC m=+46.353027175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdm9z" (UID: "0510fe39-50bf-472c-adc1-f078f5406fdd") : secret "insights-runtime-extractor-tls" not found Jul 01 19:23:58.306733 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.306712 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0510fe39-50bf-472c-adc1-f078f5406fdd-crio-socket\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.306972 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.306747 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5fd5\" (UniqueName: \"kubernetes.io/projected/0510fe39-50bf-472c-adc1-f078f5406fdd-kube-api-access-h5fd5\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.307022 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.306972 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0510fe39-50bf-472c-adc1-f078f5406fdd-data-volume\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.330189 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.330161 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5fd5\" (UniqueName: \"kubernetes.io/projected/0510fe39-50bf-472c-adc1-f078f5406fdd-kube-api-access-h5fd5\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.642315 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.642276 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:58.811354 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.811318 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0510fe39-50bf-472c-adc1-f078f5406fdd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.811517 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.811401 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.811555 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:58.811543 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 01 19:23:58.811618 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:58.811605 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls podName:0510fe39-50bf-472c-adc1-f078f5406fdd nodeName:}" failed. No retries permitted until 2026-07-01 19:23:59.811591396 +0000 UTC m=+47.357968008 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdm9z" (UID: "0510fe39-50bf-472c-adc1-f078f5406fdd") : secret "insights-runtime-extractor-tls" not found Jul 01 19:23:58.811905 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.811889 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0510fe39-50bf-472c-adc1-f078f5406fdd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:58.975577 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:58.975505 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:59.075974 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:59.075941 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:59.172804 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:59.172778 2585 scope.go:117] "RemoveContainer" containerID="197b66fc4a7bee20a6091345d94a15f2bfcef1355798b9af5b7773669388e004" Jul 01 19:23:59.173161 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:59.172916 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-m82rf_openshift-console-operator(e53cdc86-b072-4c1e-9ca3-3d504c0bc6af)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" podUID="e53cdc86-b072-4c1e-9ca3-3d504c0bc6af" Jul 01 19:23:59.570723 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:59.570682 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:59.655131 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:59.655099 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:59.753014 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:59.752979 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:23:59.821752 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:59.821679 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:23:59.821874 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:59.821825 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 01 19:23:59.821921 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:23:59.821890 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls podName:0510fe39-50bf-472c-adc1-f078f5406fdd nodeName:}" failed. No retries permitted until 2026-07-01 19:24:01.821874839 +0000 UTC m=+49.368251450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdm9z" (UID: "0510fe39-50bf-472c-adc1-f078f5406fdd") : secret "insights-runtime-extractor-tls" not found Jul 01 19:23:59.844223 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:23:59.844193 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cgn6q_a40a5624-aef1-4d71-a765-ccc60715a3d8/dns-node-resolver/0.log" Jul 01 19:24:00.177400 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:00.177368 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nnfgg" event={"ID":"9070bac0-2f2a-4fc2-b184-7363d8aa12ac","Type":"ContainerStarted","Data":"682847d9afea712de5b4047ed6e6371fba838b5e01ffb801eac48e439cf8b38b"} Jul 01 19:24:00.177728 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:00.177443 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nnfgg" event={"ID":"9070bac0-2f2a-4fc2-b184-7363d8aa12ac","Type":"ContainerStarted","Data":"3f9ce30c4fc17d18aedb1b19771ed3bffeb8143699394f75fe44d61dead6566b"} Jul 01 19:24:00.199709 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:00.199662 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nnfgg" podStartSLOduration=1.6163649260000001 podStartE2EDuration="4.199648111s" totalImagesPullingTime="2.583283185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:00.198549767 +0000 UTC m=+47.744926401" watchObservedRunningTime="2026-07-01 19:24:00.199648111 +0000 UTC m=+47.746024746" Jul 01 19:24:00.849776 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:00.849748 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-926nd_04296533-7323-4246-bcf2-fc3cd1e387ef/node-ca/0.log" Jul 01 19:24:00.914185 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:00.914150 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:01.648882 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:01.648854 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z82nd_952e2ea8-73dc-4456-9ae4-29fc1e2cdb02/serve-healthcheck-canary/0.log" Jul 01 19:24:01.839478 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:01.839442 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:24:01.839661 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:01.839597 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 01 19:24:01.839719 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:01.839670 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls podName:0510fe39-50bf-472c-adc1-f078f5406fdd nodeName:}" failed. No retries permitted until 2026-07-01 19:24:05.839644912 +0000 UTC m=+53.386021524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdm9z" (UID: "0510fe39-50bf-472c-adc1-f078f5406fdd") : secret "insights-runtime-extractor-tls" not found Jul 01 19:24:01.850869 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:01.850829 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-nnfgg_9070bac0-2f2a-4fc2-b184-7363d8aa12ac/migrator/0.log" Jul 01 19:24:02.045244 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:02.045216 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-nnfgg_9070bac0-2f2a-4fc2-b184-7363d8aa12ac/graceful-termination/0.log" Jul 01 19:24:02.247105 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:02.247045 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-685pf_5947ec3f-7e65-49cf-b511-44dc3b526ff2/kube-storage-version-migrator-operator/0.log" Jul 01 19:24:02.337746 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:02.337661 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:02.424520 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:02.424485 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:03.151305 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:03.151253 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:24:03.151305 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:03.151307 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:24:03.151861 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:03.151336 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:24:03.151861 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:03.151410 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-2n74w\" (UID: \"5168c17b-70f2-4e40-ba13-d73744c17e7a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" Jul 01 19:24:03.151861 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:03.151435 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 01 19:24:03.151861 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:03.151472 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:24:03.151861 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:03.151478 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 01 19:24:03.151861 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:03.151513 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs podName:0c94f978-1520-4f79-91df-f1f0d8048877 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:19.151493342 +0000 UTC m=+66.697869977 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs") pod "router-default-6fb847847c-xqccn" (UID: "0c94f978-1520-4f79-91df-f1f0d8048877") : secret "router-metrics-certs-default" not found Jul 01 19:24:03.151861 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:03.151480 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 01 19:24:03.151861 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:03.151436 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 19:24:03.151861 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:03.151555 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 01 19:24:03.151861 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:03.151572 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d9b4f4577-nvd9d: secret "image-registry-tls" not found Jul 01 19:24:03.151861 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:03.151577 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls podName:ef688413-3e26-4dfd-87c9-059133228994 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:19.151565862 +0000 UTC m=+66.697942475 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-gxrd5" (UID: "ef688413-3e26-4dfd-87c9-059133228994") : secret "cluster-monitoring-operator-tls" not found Jul 01 19:24:03.151861 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:03.151591 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls podName:5168c17b-70f2-4e40-ba13-d73744c17e7a nodeName:}" failed. No retries permitted until 2026-07-01 19:24:19.151585419 +0000 UTC m=+66.697962034 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-2n74w" (UID: "5168c17b-70f2-4e40-ba13-d73744c17e7a") : secret "samples-operator-tls" not found Jul 01 19:24:03.151861 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:03.151600 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls podName:9f285b32-ab0f-4888-bdc3-8c9f84a535cd nodeName:}" failed. No retries permitted until 2026-07-01 19:24:19.151595674 +0000 UTC m=+66.697972285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls") pod "dns-default-j8kx5" (UID: "9f285b32-ab0f-4888-bdc3-8c9f84a535cd") : secret "dns-default-metrics-tls" not found Jul 01 19:24:03.151861 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:03.151625 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:24:03.152334 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:03.151695 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls podName:94519212-92c3-47b1-98d5-a1dfff195e48 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:19.151689385 +0000 UTC m=+66.698065997 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls") pod "image-registry-6d9b4f4577-nvd9d" (UID: "94519212-92c3-47b1-98d5-a1dfff195e48") : secret "image-registry-tls" not found Jul 01 19:24:03.152334 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:03.151743 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle podName:0c94f978-1520-4f79-91df-f1f0d8048877 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:19.151733779 +0000 UTC m=+66.698110391 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle") pod "router-default-6fb847847c-xqccn" (UID: "0c94f978-1520-4f79-91df-f1f0d8048877") : configmap references non-existent config key: service-ca.crt Jul 01 19:24:03.252099 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:03.252050 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-rgndr\" (UID: \"9f78f69e-ffbb-4336-9976-daf5d2a9ea62\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" Jul 01 19:24:03.252256 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:03.252212 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 01 19:24:03.252295 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:03.252278 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert podName:9f78f69e-ffbb-4336-9976-daf5d2a9ea62 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:19.2522631 +0000 UTC m=+66.798639713 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-rgndr" (UID: "9f78f69e-ffbb-4336-9976-daf5d2a9ea62") : secret "networking-console-plugin-cert" not found Jul 01 19:24:03.418683 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:03.418593 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:03.516689 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:03.516653 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:04.639381 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:04.639340 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:05.202999 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:05.202964 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:05.369284 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:05.369253 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:05.626728 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:05.626692 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:05.875871 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:05.875842 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:24:05.876343 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:05.875955 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 01 19:24:05.876343 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:05.876009 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls podName:0510fe39-50bf-472c-adc1-f078f5406fdd nodeName:}" failed. No retries permitted until 2026-07-01 19:24:13.8759933 +0000 UTC m=+61.422369911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdm9z" (UID: "0510fe39-50bf-472c-adc1-f078f5406fdd") : secret "insights-runtime-extractor-tls" not found Jul 01 19:24:05.989670 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:05.989588 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:06.177296 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:06.177268 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:07.100502 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:07.100459 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:07.311446 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:07.311417 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:07.561828 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:07.561794 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:07.719787 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:07.719752 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:07.900270 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:07.900184 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:08.911725 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:08.911698 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:09.010668 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:09.010612 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:09.111626 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:09.111574 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:09.302608 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:09.302577 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:09.564533 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:09.564448 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:09.703584 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:09.703542 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:09.804468 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:09.804440 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:10.244216 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:10.244173 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:10.702860 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:10.702827 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:10.956463 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:10.956386 2585 scope.go:117] "RemoveContainer" containerID="197b66fc4a7bee20a6091345d94a15f2bfcef1355798b9af5b7773669388e004" Jul 01 19:24:11.211573 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:11.211500 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:24:11.211913 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:11.211898 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/1.log" Jul 01 19:24:11.211960 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:11.211932 2585 generic.go:356] "Generic (PLEG): container finished" podID="e53cdc86-b072-4c1e-9ca3-3d504c0bc6af" containerID="98ad79a1d5bd73a122e32134f06906cd2d5f31013685893f856834d4d106a7a9" exitCode=255 Jul 01 19:24:11.212001 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:11.211986 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" event={"ID":"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af","Type":"ContainerDied","Data":"98ad79a1d5bd73a122e32134f06906cd2d5f31013685893f856834d4d106a7a9"} Jul 01 19:24:11.212041 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:11.212030 2585 scope.go:117] "RemoveContainer" containerID="197b66fc4a7bee20a6091345d94a15f2bfcef1355798b9af5b7773669388e004" Jul 01 19:24:11.212344 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:11.212327 2585 scope.go:117] "RemoveContainer" containerID="98ad79a1d5bd73a122e32134f06906cd2d5f31013685893f856834d4d106a7a9" Jul 01 19:24:11.212516 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:11.212492 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-m82rf_openshift-console-operator(e53cdc86-b072-4c1e-9ca3-3d504c0bc6af)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" podUID="e53cdc86-b072-4c1e-9ca3-3d504c0bc6af" Jul 01 19:24:11.218437 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:11.218411 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:11.464605 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:11.464526 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:11.590433 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:11.590402 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:11.818171 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:11.818137 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:11.939762 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:11.939730 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:12.216798 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:12.216727 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:24:12.305388 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:12.305360 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:12.503287 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:12.503201 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:13.097132 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:13.097107 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frt9f" Jul 01 19:24:13.101597 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:13.101564 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:13.734462 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:13.734428 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:13.942790 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:13.942755 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:13.951580 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:13.951555 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:24:13.953763 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:13.953737 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0510fe39-50bf-472c-adc1-f078f5406fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdm9z\" (UID: \"0510fe39-50bf-472c-adc1-f078f5406fdd\") " pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:24:13.965381 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:13.965361 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5hd6c\"" Jul 01 19:24:13.973587 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:13.973572 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tdm9z" Jul 01 19:24:14.100290 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:14.100261 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:14.105568 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:14.105543 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tdm9z"] Jul 01 19:24:14.107998 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:24:14.107973 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0510fe39_50bf_472c_adc1_f078f5406fdd.slice/crio-37e4aa1c81fcb1ffa7d1530d8b826a37c4bd3a55d8e5c2c7c0fdc4b4a87de633 WatchSource:0}: Error finding container 37e4aa1c81fcb1ffa7d1530d8b826a37c4bd3a55d8e5c2c7c0fdc4b4a87de633: Status 404 returned error can't find the container with id 37e4aa1c81fcb1ffa7d1530d8b826a37c4bd3a55d8e5c2c7c0fdc4b4a87de633 Jul 01 19:24:14.223056 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:14.223021 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdm9z" event={"ID":"0510fe39-50bf-472c-adc1-f078f5406fdd","Type":"ContainerStarted","Data":"37e4aa1c81fcb1ffa7d1530d8b826a37c4bd3a55d8e5c2c7c0fdc4b4a87de633"} Jul 01 19:24:14.310345 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:14.310311 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:15.399636 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:15.399591 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:15.548281 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:15.548247 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:15.696696 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:15.696552 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:15.902184 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:15.902153 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:17.365964 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:17.365920 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:17.439507 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:17.439461 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:17.675523 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:17.675438 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:24:17.675688 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:17.675536 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:24:17.675940 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:17.675914 2585 scope.go:117] "RemoveContainer" containerID="98ad79a1d5bd73a122e32134f06906cd2d5f31013685893f856834d4d106a7a9" Jul 01 19:24:17.676158 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:17.676136 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-m82rf_openshift-console-operator(e53cdc86-b072-4c1e-9ca3-3d504c0bc6af)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" podUID="e53cdc86-b072-4c1e-9ca3-3d504c0bc6af" Jul 01 19:24:18.695599 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:18.695561 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs\") pod \"network-metrics-daemon-m2flh\" (UID: \"688cb495-bdbd-40fc-a726-7bc07d269276\") " pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:24:18.698292 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:18.698271 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 01 19:24:18.707903 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:18.707877 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/688cb495-bdbd-40fc-a726-7bc07d269276-metrics-certs\") pod \"network-metrics-daemon-m2flh\" (UID: \"688cb495-bdbd-40fc-a726-7bc07d269276\") " pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:24:18.796711 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:18.796676 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpbv\" (UniqueName: \"kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv\") pod \"network-check-target-csgsb\" (UID: \"54990c60-95a2-4a2c-9d4d-690c9994707b\") " pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:24:18.796858 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:18.796737 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:24:18.799007 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:18.798985 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzpbv\" (UniqueName: \"kubernetes.io/projected/54990c60-95a2-4a2c-9d4d-690c9994707b-kube-api-access-wzpbv\") pod \"network-check-target-csgsb\" (UID: \"54990c60-95a2-4a2c-9d4d-690c9994707b\") " pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:24:18.799203 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:18.799184 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 01 19:24:18.809049 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:18.809029 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2553301-dbf4-4594-9dea-5d6f3b6ff8f2-original-pull-secret\") pod \"global-pull-secret-syncer-qsrzd\" (UID: \"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2\") " pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:24:18.868804 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:18.868770 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-64bmf\"" Jul 01 19:24:18.876774 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:18.876757 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:24:18.978799 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:18.978774 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4nv7g\"" Jul 01 19:24:18.986652 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:18.986626 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2flh" Jul 01 19:24:18.988566 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:18.988544 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsrzd" Jul 01 19:24:18.996972 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:18.996943 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-csgsb"] Jul 01 19:24:19.001222 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:24:19.001196 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54990c60_95a2_4a2c_9d4d_690c9994707b.slice/crio-bcef0682100e83201f8d9d8d379be71289287765a7bd374041d5eec8dab9db8b WatchSource:0}: Error finding container bcef0682100e83201f8d9d8d379be71289287765a7bd374041d5eec8dab9db8b: Status 404 returned error can't find the container with id bcef0682100e83201f8d9d8d379be71289287765a7bd374041d5eec8dab9db8b Jul 01 19:24:19.120740 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.120717 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m2flh"] Jul 01 19:24:19.123105 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:24:19.123077 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688cb495_bdbd_40fc_a726_7bc07d269276.slice/crio-609f5404214891a9179b04086a6f1ade4d29c54af5fb98e060054b13c423ad44 WatchSource:0}: Error finding container 609f5404214891a9179b04086a6f1ade4d29c54af5fb98e060054b13c423ad44: Status 404 returned error can't find the container with id 609f5404214891a9179b04086a6f1ade4d29c54af5fb98e060054b13c423ad44 Jul 01 19:24:19.139322 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.139297 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qsrzd"] Jul 01 19:24:19.141918 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:24:19.141895 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2553301_dbf4_4594_9dea_5d6f3b6ff8f2.slice/crio-87c8860786f66831ace9691ca457c70756cabcea2af6846f2013a1dd2ca2318f WatchSource:0}: Error finding container 87c8860786f66831ace9691ca457c70756cabcea2af6846f2013a1dd2ca2318f: Status 404 returned error can't find the container with id 87c8860786f66831ace9691ca457c70756cabcea2af6846f2013a1dd2ca2318f Jul 01 19:24:19.200877 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.200804 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:24:19.200877 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.200860 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:24:19.201095 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.200942 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:24:19.201095 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.200972 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:24:19.201095 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.200998 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:24:19.201095 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.201029 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-2n74w\" (UID: \"5168c17b-70f2-4e40-ba13-d73744c17e7a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" Jul 01 19:24:19.201715 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.201661 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c94f978-1520-4f79-91df-f1f0d8048877-service-ca-bundle\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:24:19.203917 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.203578 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5168c17b-70f2-4e40-ba13-d73744c17e7a-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-2n74w\" (UID: \"5168c17b-70f2-4e40-ba13-d73744c17e7a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" Jul 01 19:24:19.203917 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.203707 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f285b32-ab0f-4888-bdc3-8c9f84a535cd-metrics-tls\") pod \"dns-default-j8kx5\" (UID: \"9f285b32-ab0f-4888-bdc3-8c9f84a535cd\") " pod="openshift-dns/dns-default-j8kx5" Jul 01 19:24:19.203917 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.203761 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c94f978-1520-4f79-91df-f1f0d8048877-metrics-certs\") pod \"router-default-6fb847847c-xqccn\" (UID: \"0c94f978-1520-4f79-91df-f1f0d8048877\") " pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:24:19.204269 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.203920 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef688413-3e26-4dfd-87c9-059133228994-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-gxrd5\" (UID: \"ef688413-3e26-4dfd-87c9-059133228994\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:24:19.204269 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.204013 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls\") pod \"image-registry-6d9b4f4577-nvd9d\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:24:19.244368 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.244329 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdm9z" event={"ID":"0510fe39-50bf-472c-adc1-f078f5406fdd","Type":"ContainerStarted","Data":"629274e5e6d881a343014a032b70ccd0b2e26b23df5098ffcaf11c138974b8eb"} Jul 01 19:24:19.244508 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.244400 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdm9z" event={"ID":"0510fe39-50bf-472c-adc1-f078f5406fdd","Type":"ContainerStarted","Data":"2e7b9e9adb2fc1b8b8454b506e745f236d913e66338b4908fddd403c1eb087e8"} Jul 01 19:24:19.245927 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.245894 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-csgsb" event={"ID":"54990c60-95a2-4a2c-9d4d-690c9994707b","Type":"ContainerStarted","Data":"53843f076deb6b7685a3c52a39e191fce9dad4b810420b5690fdb3b5d862759b"} Jul 01 19:24:19.246078 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.245944 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-csgsb" event={"ID":"54990c60-95a2-4a2c-9d4d-690c9994707b","Type":"ContainerStarted","Data":"bcef0682100e83201f8d9d8d379be71289287765a7bd374041d5eec8dab9db8b"} Jul 01 19:24:19.246078 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.245986 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:24:19.247374 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.247343 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qsrzd" event={"ID":"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2","Type":"ContainerStarted","Data":"87c8860786f66831ace9691ca457c70756cabcea2af6846f2013a1dd2ca2318f"} Jul 01 19:24:19.248605 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.248581 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m2flh" event={"ID":"688cb495-bdbd-40fc-a726-7bc07d269276","Type":"ContainerStarted","Data":"609f5404214891a9179b04086a6f1ade4d29c54af5fb98e060054b13c423ad44"} Jul 01 19:24:19.263450 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.263407 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-csgsb" podStartSLOduration=66.263394348 podStartE2EDuration="1m6.263394348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:19.262506473 +0000 UTC m=+66.808883109" watchObservedRunningTime="2026-07-01 19:24:19.263394348 +0000 UTC m=+66.809771003" Jul 01 19:24:19.302111 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.302080 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-rgndr\" (UID: \"9f78f69e-ffbb-4336-9976-daf5d2a9ea62\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" Jul 01 19:24:19.304935 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.304914 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f78f69e-ffbb-4336-9976-daf5d2a9ea62-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-rgndr\" (UID: \"9f78f69e-ffbb-4336-9976-daf5d2a9ea62\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" Jul 01 19:24:19.376579 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.376550 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nzzx8\"" Jul 01 19:24:19.384658 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.384634 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:24:19.402953 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.402750 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lm6s9\"" Jul 01 19:24:19.410654 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.410632 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" Jul 01 19:24:19.472191 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.472166 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tqsg9\"" Jul 01 19:24:19.480391 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.480157 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:24:19.483782 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.483344 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9p2w7\"" Jul 01 19:24:19.489150 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.489048 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gt8f8\"" Jul 01 19:24:19.492231 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.491961 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j8kx5" Jul 01 19:24:19.497544 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.497357 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" Jul 01 19:24:19.515429 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.515401 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-zcpfc\"" Jul 01 19:24:19.523543 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.523287 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" Jul 01 19:24:19.545438 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.545365 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d9b4f4577-nvd9d"] Jul 01 19:24:19.554006 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:24:19.553968 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94519212_92c3_47b1_98d5_a1dfff195e48.slice/crio-f3dd7eacaf90c871d61c4843c216d1ecdc35ce34d05ec96240cff659341e29a9 WatchSource:0}: Error finding container f3dd7eacaf90c871d61c4843c216d1ecdc35ce34d05ec96240cff659341e29a9: Status 404 returned error can't find the container with id f3dd7eacaf90c871d61c4843c216d1ecdc35ce34d05ec96240cff659341e29a9 Jul 01 19:24:19.587758 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.587639 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w"] Jul 01 19:24:19.676715 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.676267 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6fb847847c-xqccn"] Jul 01 19:24:19.696158 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.696081 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5"] Jul 01 19:24:19.703419 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:24:19.703387 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef688413_3e26_4dfd_87c9_059133228994.slice/crio-bae89ae5a97c311a5971032367b2a4bf087524181596e0b6f385c863c096eec3 WatchSource:0}: Error finding container bae89ae5a97c311a5971032367b2a4bf087524181596e0b6f385c863c096eec3: Status 404 returned error can't find the container with id bae89ae5a97c311a5971032367b2a4bf087524181596e0b6f385c863c096eec3 Jul 01 19:24:19.735704 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.735678 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr"] Jul 01 19:24:19.738518 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:24:19.738490 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f78f69e_ffbb_4336_9976_daf5d2a9ea62.slice/crio-7a9506dd6f1d44f06eb17174c736f8df9187042058c955a66bc8adf67893124b WatchSource:0}: Error finding container 7a9506dd6f1d44f06eb17174c736f8df9187042058c955a66bc8adf67893124b: Status 404 returned error can't find the container with id 7a9506dd6f1d44f06eb17174c736f8df9187042058c955a66bc8adf67893124b Jul 01 19:24:19.906863 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:19.904512 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j8kx5"] Jul 01 19:24:20.255653 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:20.255617 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" event={"ID":"9f78f69e-ffbb-4336-9976-daf5d2a9ea62","Type":"ContainerStarted","Data":"7a9506dd6f1d44f06eb17174c736f8df9187042058c955a66bc8adf67893124b"} Jul 01 19:24:20.257307 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:20.257280 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j8kx5" event={"ID":"9f285b32-ab0f-4888-bdc3-8c9f84a535cd","Type":"ContainerStarted","Data":"2421c55e6f9804278b06332ee37ddcfbd8ffcf8b3332f105b192ee376cabfb79"} Jul 01 19:24:20.259540 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:20.259498 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6fb847847c-xqccn" event={"ID":"0c94f978-1520-4f79-91df-f1f0d8048877","Type":"ContainerStarted","Data":"bfbf126bd19fccd61563b92ab5c606110d5ca3d95f354a40373a42b288704e93"} Jul 01 19:24:20.259646 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:20.259541 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6fb847847c-xqccn" event={"ID":"0c94f978-1520-4f79-91df-f1f0d8048877","Type":"ContainerStarted","Data":"7b700d0321a7978b8dbe766d857c9f0f18b7559083406aa69e04019b1399ab25"} Jul 01 19:24:20.261153 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:20.261104 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" event={"ID":"ef688413-3e26-4dfd-87c9-059133228994","Type":"ContainerStarted","Data":"bae89ae5a97c311a5971032367b2a4bf087524181596e0b6f385c863c096eec3"} Jul 01 19:24:20.264041 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:20.263403 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" event={"ID":"94519212-92c3-47b1-98d5-a1dfff195e48","Type":"ContainerStarted","Data":"7155b15f270aa13ae4c86fed8847e561411d9001572ec66c99457d9c9268dfa8"} Jul 01 19:24:20.264041 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:20.263447 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" event={"ID":"94519212-92c3-47b1-98d5-a1dfff195e48","Type":"ContainerStarted","Data":"f3dd7eacaf90c871d61c4843c216d1ecdc35ce34d05ec96240cff659341e29a9"} Jul 01 19:24:20.264041 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:20.263986 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:24:20.266725 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:20.266660 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" event={"ID":"5168c17b-70f2-4e40-ba13-d73744c17e7a","Type":"ContainerStarted","Data":"7adcecd3a1c4a91b7f07573134767592285b6cf6ef3c6f02690c8daf2f4950a8"} Jul 01 19:24:20.300942 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:20.300042 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6fb847847c-xqccn" podStartSLOduration=50.300021728 podStartE2EDuration="50.300021728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:20.29966674 +0000 UTC m=+67.846043394" watchObservedRunningTime="2026-07-01 19:24:20.300021728 +0000 UTC m=+67.846398363" Jul 01 19:24:20.481464 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:20.481389 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:24:20.484435 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:20.484411 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:24:20.519382 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:20.519278 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" podStartSLOduration=67.519252541 podStartE2EDuration="1m7.519252541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:20.325868351 +0000 UTC m=+67.872244986" watchObservedRunningTime="2026-07-01 19:24:20.519252541 +0000 UTC m=+68.065629175" Jul 01 19:24:21.271165 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:21.271106 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:24:21.272866 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:21.272767 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6fb847847c-xqccn" Jul 01 19:24:21.451966 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:21.451931 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:21.659496 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:21.659416 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:21.758358 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:21.758304 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:24.933865 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:24.933822 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:25.033564 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.033525 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:25.094275 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.094244 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-2njh6"] Jul 01 19:24:25.109203 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.109165 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:25.117159 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.117120 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-2njh6"] Jul 01 19:24:25.117301 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.117262 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-2njh6" Jul 01 19:24:25.119635 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.119614 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xb4gl\"" Jul 01 19:24:25.119729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.119635 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 01 19:24:25.155392 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.155358 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/10e4a2f9-1c4b-41b4-bb46-feb2922b4b26-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-2njh6\" (UID: \"10e4a2f9-1c4b-41b4-bb46-feb2922b4b26\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-2njh6" Jul 01 19:24:25.234411 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.234299 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:25.256145 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.256106 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/10e4a2f9-1c4b-41b4-bb46-feb2922b4b26-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-2njh6\" (UID: \"10e4a2f9-1c4b-41b4-bb46-feb2922b4b26\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-2njh6" Jul 01 19:24:25.256297 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:25.256276 2585 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 01 19:24:25.256373 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:25.256349 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10e4a2f9-1c4b-41b4-bb46-feb2922b4b26-tls-certificates podName:10e4a2f9-1c4b-41b4-bb46-feb2922b4b26 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:25.756328038 +0000 UTC m=+73.302704702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/10e4a2f9-1c4b-41b4-bb46-feb2922b4b26-tls-certificates") pod "prometheus-operator-admission-webhook-7ddc8784d4-2njh6" (UID: "10e4a2f9-1c4b-41b4-bb46-feb2922b4b26") : secret "prometheus-operator-admission-webhook-tls" not found Jul 01 19:24:25.295138 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.295104 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" event={"ID":"ef688413-3e26-4dfd-87c9-059133228994","Type":"ContainerStarted","Data":"d78ea6db750a82608a55f38735fc79dfff0a9e5fa0f4ceff011c609b83a846a1"} Jul 01 19:24:25.297681 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.297646 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdm9z" event={"ID":"0510fe39-50bf-472c-adc1-f078f5406fdd","Type":"ContainerStarted","Data":"9c0abfec625cb6991303e3dddaf8b3b1b05427be147c4d7df5f2e60cc1b9e2f4"} Jul 01 19:24:25.299332 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.299303 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m2flh" event={"ID":"688cb495-bdbd-40fc-a726-7bc07d269276","Type":"ContainerStarted","Data":"5d88f9c2eafa802dea6ac8972083ab87e302a5fb9f3df65e9b93584f12964ba5"} Jul 01 19:24:25.319614 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.319567 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-gxrd5" podStartSLOduration=50.479276656 podStartE2EDuration="55.319552296s" totalImagesPullingTime="4.84027564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:25.318683426 +0000 UTC m=+72.865060069" watchObservedRunningTime="2026-07-01 19:24:25.319552296 +0000 UTC m=+72.865928929" Jul 01 19:24:25.332829 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.332803 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:25.347506 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.347467 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tdm9z" podStartSLOduration=17.013436584 podStartE2EDuration="27.34745157s" totalImagesPullingTime="10.334014986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:23:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:25.345789013 +0000 UTC m=+72.892165647" watchObservedRunningTime="2026-07-01 19:24:25.34745157 +0000 UTC m=+72.893828221" Jul 01 19:24:25.526915 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.526870 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:25.761203 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.761168 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/10e4a2f9-1c4b-41b4-bb46-feb2922b4b26-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-2njh6\" (UID: \"10e4a2f9-1c4b-41b4-bb46-feb2922b4b26\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-2njh6" Jul 01 19:24:25.764080 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:25.764038 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/10e4a2f9-1c4b-41b4-bb46-feb2922b4b26-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-2njh6\" (UID: \"10e4a2f9-1c4b-41b4-bb46-feb2922b4b26\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-2njh6" Jul 01 19:24:26.027551 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:26.027520 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-2njh6" Jul 01 19:24:26.474024 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:26.473340 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-2njh6"] Jul 01 19:24:26.475857 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:24:26.475827 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e4a2f9_1c4b_41b4_bb46_feb2922b4b26.slice/crio-6cb06c7b0a5a5162bfecfb19326cbfc29f5825e339ef2ea7f176a3e5166c5afc WatchSource:0}: Error finding container 6cb06c7b0a5a5162bfecfb19326cbfc29f5825e339ef2ea7f176a3e5166c5afc: Status 404 returned error can't find the container with id 6cb06c7b0a5a5162bfecfb19326cbfc29f5825e339ef2ea7f176a3e5166c5afc Jul 01 19:24:27.310786 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.310745 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m2flh" event={"ID":"688cb495-bdbd-40fc-a726-7bc07d269276","Type":"ContainerStarted","Data":"1226d5707dff8566ec752cdb3d394ed1414766aa6d691e8b051780f6235523b9"} Jul 01 19:24:27.312390 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.312362 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" event={"ID":"9f78f69e-ffbb-4336-9976-daf5d2a9ea62","Type":"ContainerStarted","Data":"2e758a90322fa0949e4c02ee24cd86fa6f12fcf082af796f37cae2d9150033fa"} Jul 01 19:24:27.314626 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.314599 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j8kx5" event={"ID":"9f285b32-ab0f-4888-bdc3-8c9f84a535cd","Type":"ContainerStarted","Data":"d7d8ecfbc8932122645cc0bee4e4569ab00d5682f634fd5b6afd678ec96eb3fa"} Jul 01 19:24:27.314713 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.314645 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j8kx5" event={"ID":"9f285b32-ab0f-4888-bdc3-8c9f84a535cd","Type":"ContainerStarted","Data":"ca01cfc9856f19af957b115f55b3923c53300d7819acbb7e0f27f79bc57538b2"} Jul 01 19:24:27.314713 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.314693 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j8kx5" Jul 01 19:24:27.316443 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.316393 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qsrzd" event={"ID":"e2553301-dbf4-4594-9dea-5d6f3b6ff8f2","Type":"ContainerStarted","Data":"332f7f390ecde64507766bd51971edb4d802d72f46afe11d564073b2b16538c8"} Jul 01 19:24:27.318336 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.318313 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" event={"ID":"5168c17b-70f2-4e40-ba13-d73744c17e7a","Type":"ContainerStarted","Data":"2fcab6d6764c7c73ce5524eef972d1342d3df1ac2b869ff4722b86f1d13c820a"} Jul 01 19:24:27.318422 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.318360 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" event={"ID":"5168c17b-70f2-4e40-ba13-d73744c17e7a","Type":"ContainerStarted","Data":"f76420edee1b2988c3151784ceb3729788eabad8b92161dd2b0c27bfb3292ebd"} Jul 01 19:24:27.319507 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.319487 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-2njh6" event={"ID":"10e4a2f9-1c4b-41b4-bb46-feb2922b4b26","Type":"ContainerStarted","Data":"6cb06c7b0a5a5162bfecfb19326cbfc29f5825e339ef2ea7f176a3e5166c5afc"} Jul 01 19:24:27.329437 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.329391 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m2flh" podStartSLOduration=68.909193925 podStartE2EDuration="1m14.329373482s" totalImagesPullingTime="5.420179557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:27.32809049 +0000 UTC m=+74.874467122" watchObservedRunningTime="2026-07-01 19:24:27.329373482 +0000 UTC m=+74.875750116" Jul 01 19:24:27.345734 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.345683 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qsrzd" podStartSLOduration=67.218258698 podStartE2EDuration="1m14.345668275s" totalImagesPullingTime="7.127409577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:27.345179446 +0000 UTC m=+74.891556080" watchObservedRunningTime="2026-07-01 19:24:27.345668275 +0000 UTC m=+74.892044903" Jul 01 19:24:27.362898 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.362856 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-2n74w" podStartSLOduration=50.737564794 podStartE2EDuration="57.362843947s" totalImagesPullingTime="6.625279153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:27.361652137 +0000 UTC m=+74.908028772" watchObservedRunningTime="2026-07-01 19:24:27.362843947 +0000 UTC m=+74.909220581" Jul 01 19:24:27.382573 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.382530 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-j8kx5" podStartSLOduration=33.980077175 podStartE2EDuration="40.382512392s" totalImagesPullingTime="6.402435217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:27.381827157 +0000 UTC m=+74.928203805" watchObservedRunningTime="2026-07-01 19:24:27.382512392 +0000 UTC m=+74.928889029" Jul 01 19:24:27.404185 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.404138 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-rgndr" podStartSLOduration=41.596737567 podStartE2EDuration="46.404125421s" totalImagesPullingTime="4.807387854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:27.403139751 +0000 UTC m=+74.949516384" watchObservedRunningTime="2026-07-01 19:24:27.404125421 +0000 UTC m=+74.950502053" Jul 01 19:24:27.476427 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.476397 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:27.556221 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.556180 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:27.653559 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.653471 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:27.753865 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.753832 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:27.821669 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.821635 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:27.980634 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:27.980552 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:28.325019 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:28.324987 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-2njh6" event={"ID":"10e4a2f9-1c4b-41b4-bb46-feb2922b4b26","Type":"ContainerStarted","Data":"2fb4037d30136ed77f3f6bf6502b540100f97ffae08c4b63f952b3f8eb1b05a0"} Jul 01 19:24:28.340831 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:28.340778 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-2njh6" podStartSLOduration=1.6122089750000002 podStartE2EDuration="3.340757239s" totalImagesPullingTime="1.728548264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:28.34062124 +0000 UTC m=+75.886997893" watchObservedRunningTime="2026-07-01 19:24:28.340757239 +0000 UTC m=+75.887133874" Jul 01 19:24:29.329567 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:29.329532 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-2njh6" Jul 01 19:24:29.334302 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:29.334279 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-2njh6" Jul 01 19:24:30.122285 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:30.122250 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:30.193009 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:30.192973 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:30.420837 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:30.420754 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:32.384699 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:32.384659 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:32.489620 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:32.489566 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:32.578001 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:32.577953 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:32.709491 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:32.709395 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:32.808092 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:32.808048 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:32.957703 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:32.957669 2585 scope.go:117] "RemoveContainer" containerID="98ad79a1d5bd73a122e32134f06906cd2d5f31013685893f856834d4d106a7a9" Jul 01 19:24:33.346338 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:33.346312 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:24:33.346510 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:33.346416 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" event={"ID":"e53cdc86-b072-4c1e-9ca3-3d504c0bc6af","Type":"ContainerStarted","Data":"ccafad3de0b3ead1cba55b4497e33ae3957d5c34acda01da00b66c7c2f4f3388"} Jul 01 19:24:33.346828 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:33.346798 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:24:33.364247 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:33.364202 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" podStartSLOduration=55.869183687 podStartE2EDuration="1m3.364186721s" totalImagesPullingTime="7.495003034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:33.363824145 +0000 UTC m=+80.910200782" watchObservedRunningTime="2026-07-01 19:24:33.364186721 +0000 UTC m=+80.910563356" Jul 01 19:24:33.865035 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:33.865004 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-m82rf" Jul 01 19:24:34.068595 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:34.068565 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:34.183824 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:34.183742 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:34.283934 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:34.283906 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:34.348849 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:34.348814 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:34.524533 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:34.524502 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:34.779169 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:34.779085 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:34.840001 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:34.839967 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:34.921768 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:34.921735 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:35.022059 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:35.022028 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:35.128728 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:35.128647 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:35.243309 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:35.243263 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:35.740701 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:35.740671 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:35.956107 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:35.956055 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:36.398617 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:36.398588 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:37.146270 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.146231 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:37.258661 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.258627 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:37.327590 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.327564 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j8kx5" Jul 01 19:24:37.371802 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.371737 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:37.480904 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.480821 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:37.603030 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.602966 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:37.632507 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.632478 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dj7cs"] Jul 01 19:24:37.635575 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.635560 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.639270 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.639248 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 01 19:24:37.639591 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.639555 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bfvnn\"" Jul 01 19:24:37.639591 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.639580 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 01 19:24:37.639761 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.639641 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 01 19:24:37.639761 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.639653 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 01 19:24:37.722599 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.722540 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:37.761648 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.761581 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-textfile\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.761648 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.761624 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-wtmp\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.761648 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.761646 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd873e60-27ca-4f25-a45a-54a66c46fc5a-sys\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.761860 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.761665 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-accelerators-collector-config\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.761860 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.761729 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-tls\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.761860 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.761759 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffmjt\" (UniqueName: \"kubernetes.io/projected/cd873e60-27ca-4f25-a45a-54a66c46fc5a-kube-api-access-ffmjt\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.761860 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.761786 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd873e60-27ca-4f25-a45a-54a66c46fc5a-metrics-client-ca\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.761860 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.761808 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd873e60-27ca-4f25-a45a-54a66c46fc5a-root\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.762049 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.761883 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.863215 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.863177 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-wtmp\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.863394 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.863222 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd873e60-27ca-4f25-a45a-54a66c46fc5a-sys\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.863394 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.863282 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd873e60-27ca-4f25-a45a-54a66c46fc5a-sys\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.863394 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.863322 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-accelerators-collector-config\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.863394 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.863348 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-wtmp\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.863394 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.863358 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-tls\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.863394 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.863395 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffmjt\" (UniqueName: \"kubernetes.io/projected/cd873e60-27ca-4f25-a45a-54a66c46fc5a-kube-api-access-ffmjt\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.863689 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.863424 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd873e60-27ca-4f25-a45a-54a66c46fc5a-metrics-client-ca\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.863689 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.863444 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd873e60-27ca-4f25-a45a-54a66c46fc5a-root\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.863689 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:37.863449 2585 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 01 19:24:37.863689 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.863487 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.863689 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:24:37.863508 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-tls podName:cd873e60-27ca-4f25-a45a-54a66c46fc5a nodeName:}" failed. No retries permitted until 2026-07-01 19:24:38.363488078 +0000 UTC m=+85.909864702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-tls") pod "node-exporter-dj7cs" (UID: "cd873e60-27ca-4f25-a45a-54a66c46fc5a") : secret "node-exporter-tls" not found Jul 01 19:24:37.863689 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.863540 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd873e60-27ca-4f25-a45a-54a66c46fc5a-root\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.863689 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.863551 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-textfile\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.863990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.863846 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-textfile\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.863990 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.863935 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-accelerators-collector-config\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.864121 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.864104 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd873e60-27ca-4f25-a45a-54a66c46fc5a-metrics-client-ca\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.865726 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.865711 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.894799 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.894760 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffmjt\" (UniqueName: \"kubernetes.io/projected/cd873e60-27ca-4f25-a45a-54a66c46fc5a-kube-api-access-ffmjt\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:37.910732 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:37.910707 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:38.143735 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:38.143700 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:38.233757 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:38.233723 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:38.367962 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:38.367931 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-tls\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:38.370757 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:38.370729 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd873e60-27ca-4f25-a45a-54a66c46fc5a-node-exporter-tls\") pod \"node-exporter-dj7cs\" (UID: \"cd873e60-27ca-4f25-a45a-54a66c46fc5a\") " pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:38.374546 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:38.374523 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:38.473895 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:38.473808 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:38.545222 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:38.545196 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dj7cs" Jul 01 19:24:38.555559 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:24:38.555536 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd873e60_27ca_4f25_a45a_54a66c46fc5a.slice/crio-d59f2bd1ae439066ca41b558105b6b79b64daf7621ed70643c586843ecc5bf38 WatchSource:0}: Error finding container d59f2bd1ae439066ca41b558105b6b79b64daf7621ed70643c586843ecc5bf38: Status 404 returned error can't find the container with id d59f2bd1ae439066ca41b558105b6b79b64daf7621ed70643c586843ecc5bf38 Jul 01 19:24:38.725617 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:38.725528 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:38.852827 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:38.852792 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:38.988025 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:38.987937 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:39.372137 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:39.372097 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dj7cs" event={"ID":"cd873e60-27ca-4f25-a45a-54a66c46fc5a","Type":"ContainerStarted","Data":"d59f2bd1ae439066ca41b558105b6b79b64daf7621ed70643c586843ecc5bf38"} Jul 01 19:24:39.434669 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:39.434622 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:39.539298 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:39.539262 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:40.377507 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:40.377474 2585 generic.go:356] "Generic (PLEG): container finished" podID="cd873e60-27ca-4f25-a45a-54a66c46fc5a" containerID="8d3278921bb41309bf48083235a81b5e9a8ed1cc4037efebf9452ee1ec3e3628" exitCode=0 Jul 01 19:24:40.377909 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:40.377554 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dj7cs" event={"ID":"cd873e60-27ca-4f25-a45a-54a66c46fc5a","Type":"ContainerDied","Data":"8d3278921bb41309bf48083235a81b5e9a8ed1cc4037efebf9452ee1ec3e3628"} Jul 01 19:24:40.610572 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:40.610540 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:40.698239 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:40.698157 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:40.787568 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:40.787530 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:40.884143 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:40.884108 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:41.383697 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:41.383658 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dj7cs" event={"ID":"cd873e60-27ca-4f25-a45a-54a66c46fc5a","Type":"ContainerStarted","Data":"974b40c9ace020022b9e4ab05c2e655c46f51575f3e95671b24b8beaf3ec787c"} Jul 01 19:24:41.384095 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:41.383718 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dj7cs" event={"ID":"cd873e60-27ca-4f25-a45a-54a66c46fc5a","Type":"ContainerStarted","Data":"447179b734128c5e5ef53178d08493b73645700bcb24826c9ec69ac14ff3e070"} Jul 01 19:24:41.409976 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:41.407210 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dj7cs" podStartSLOduration=3.342709247 podStartE2EDuration="4.40718629s" totalImagesPullingTime="998.916302ms" totalInitContainerRuntime="65.560741ms" isStatefulPod=true podCreationTimestamp="2026-07-01 19:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:41.404601222 +0000 UTC m=+88.950977857" watchObservedRunningTime="2026-07-01 19:24:41.40718629 +0000 UTC m=+88.953562925" Jul 01 19:24:42.281318 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:42.281292 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:24:42.329719 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:42.329671 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:42.403032 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:42.402999 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:42.609364 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:42.609284 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:42.692685 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:42.692655 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:43.800540 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:43.800503 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:43.898577 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:43.898538 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:44.005143 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:44.005111 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:44.226626 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:44.226553 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d9b4f4577-nvd9d"] Jul 01 19:24:44.379200 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:44.379169 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:44.496320 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:44.496239 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:44.731780 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:44.731752 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:44.808991 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:44.808963 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:48.304299 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:48.304257 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:48.573427 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:48.573341 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:48.675215 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:48.675181 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:49.809270 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:49.809240 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:49.871854 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:49.871821 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:50.269898 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:50.269822 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-csgsb" Jul 01 19:24:51.150922 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.150890 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dcc9bb7fd-znq6q"] Jul 01 19:24:51.172754 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.172722 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:51.203568 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.203535 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dcc9bb7fd-znq6q"] Jul 01 19:24:51.203728 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.203625 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.206329 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.206306 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 01 19:24:51.206451 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.206306 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 01 19:24:51.206451 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.206317 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2twc4\"" Jul 01 19:24:51.206451 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.206317 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 01 19:24:51.206950 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.206931 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 01 19:24:51.206950 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.206958 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 01 19:24:51.207174 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.206997 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 01 19:24:51.207574 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.207559 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 01 19:24:51.213904 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.213885 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 01 19:24:51.241389 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.241364 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:51.378231 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.378200 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-oauth-serving-cert\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.378413 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.378250 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adda2008-79e0-420b-a539-924ce999e7d9-console-serving-cert\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.378413 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.378302 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxxnq\" (UniqueName: \"kubernetes.io/projected/adda2008-79e0-420b-a539-924ce999e7d9-kube-api-access-nxxnq\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.378413 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.378350 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adda2008-79e0-420b-a539-924ce999e7d9-console-oauth-config\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.378413 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.378368 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-trusted-ca-bundle\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.378567 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.378428 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-console-config\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.378567 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.378462 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-service-ca\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.479910 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.479821 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adda2008-79e0-420b-a539-924ce999e7d9-console-oauth-config\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.479910 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.479859 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-trusted-ca-bundle\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.479910 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.479888 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-console-config\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.479910 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.479916 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-service-ca\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.480252 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.479935 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-oauth-serving-cert\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.480252 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.479958 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adda2008-79e0-420b-a539-924ce999e7d9-console-serving-cert\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.480252 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.479975 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxnq\" (UniqueName: \"kubernetes.io/projected/adda2008-79e0-420b-a539-924ce999e7d9-kube-api-access-nxxnq\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.480746 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.480718 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-service-ca\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.480843 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.480781 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-oauth-serving-cert\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.480888 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.480863 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-trusted-ca-bundle\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.480922 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.480863 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-console-config\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.482337 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.482315 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adda2008-79e0-420b-a539-924ce999e7d9-console-serving-cert\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.482420 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.482351 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adda2008-79e0-420b-a539-924ce999e7d9-console-oauth-config\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.488394 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.488374 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxnq\" (UniqueName: \"kubernetes.io/projected/adda2008-79e0-420b-a539-924ce999e7d9-kube-api-access-nxxnq\") pod \"console-dcc9bb7fd-znq6q\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.518885 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.518857 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:24:51.657421 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:51.657398 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dcc9bb7fd-znq6q"] Jul 01 19:24:51.659496 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:24:51.659464 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadda2008_79e0_420b_a539_924ce999e7d9.slice/crio-43bd8658205c5edd6c474fac33fda1a363ecd684e3fe68cc1b1a35e83548f580 WatchSource:0}: Error finding container 43bd8658205c5edd6c474fac33fda1a363ecd684e3fe68cc1b1a35e83548f580: Status 404 returned error can't find the container with id 43bd8658205c5edd6c474fac33fda1a363ecd684e3fe68cc1b1a35e83548f580 Jul 01 19:24:52.157395 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:52.157348 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:52.426970 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:52.426890 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcc9bb7fd-znq6q" event={"ID":"adda2008-79e0-420b-a539-924ce999e7d9","Type":"ContainerStarted","Data":"43bd8658205c5edd6c474fac33fda1a363ecd684e3fe68cc1b1a35e83548f580"} Jul 01 19:24:55.440804 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:55.440698 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcc9bb7fd-znq6q" event={"ID":"adda2008-79e0-420b-a539-924ce999e7d9","Type":"ContainerStarted","Data":"dc61952097755215287ee2d418f8678d6a8524cff0df57d329b97859f5a54298"} Jul 01 19:24:55.459031 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:55.458983 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dcc9bb7fd-znq6q" podStartSLOduration=1.085852415 podStartE2EDuration="4.458967621s" totalImagesPullingTime="3.373115206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:55.457685305 +0000 UTC m=+103.004061938" watchObservedRunningTime="2026-07-01 19:24:55.458967621 +0000 UTC m=+103.005344255" Jul 01 19:24:57.312294 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:57.312256 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:57.376081 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:57.376041 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:58.928005 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:58.927959 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:59.033540 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:59.033506 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:59.125824 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:59.125786 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:59.226222 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:59.226131 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:24:59.955179 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:59.955150 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7584cf5bc7-lbzxm"] Jul 01 19:24:59.991151 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:59.991119 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7584cf5bc7-lbzxm"] Jul 01 19:24:59.991305 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:24:59.991255 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.052692 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.052663 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-trusted-ca-bundle\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.052884 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.052713 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-service-ca\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.052884 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.052751 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-config\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.052884 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.052771 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tntw9\" (UniqueName: \"kubernetes.io/projected/4852b307-ba58-41ee-a079-ba1a7d2e3484-kube-api-access-tntw9\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.052884 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.052792 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-oauth-serving-cert\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.053040 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.052882 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-oauth-config\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.053040 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.052937 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-serving-cert\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.154368 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.154333 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-oauth-config\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.154551 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.154379 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-serving-cert\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.154551 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.154404 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-trusted-ca-bundle\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.154551 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.154427 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-service-ca\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.154551 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.154538 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-config\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.154740 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.154576 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tntw9\" (UniqueName: \"kubernetes.io/projected/4852b307-ba58-41ee-a079-ba1a7d2e3484-kube-api-access-tntw9\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.154740 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.154610 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-oauth-serving-cert\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.155205 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.155184 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-service-ca\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.155321 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.155268 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-oauth-serving-cert\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.155402 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.155379 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-config\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.155478 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.155454 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-trusted-ca-bundle\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.157015 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.156994 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-oauth-config\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.157160 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.157143 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-serving-cert\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.163954 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.163935 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tntw9\" (UniqueName: \"kubernetes.io/projected/4852b307-ba58-41ee-a079-ba1a7d2e3484-kube-api-access-tntw9\") pod \"console-7584cf5bc7-lbzxm\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.300869 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.300831 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:00.432584 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.432351 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7584cf5bc7-lbzxm"] Jul 01 19:25:00.435185 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:25:00.435158 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4852b307_ba58_41ee_a079_ba1a7d2e3484.slice/crio-6aa67f7da275b84b1bc1dbcaea7457108d5ac762a7dffa4494c071a70c172a0f WatchSource:0}: Error finding container 6aa67f7da275b84b1bc1dbcaea7457108d5ac762a7dffa4494c071a70c172a0f: Status 404 returned error can't find the container with id 6aa67f7da275b84b1bc1dbcaea7457108d5ac762a7dffa4494c071a70c172a0f Jul 01 19:25:00.460868 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:00.460838 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7584cf5bc7-lbzxm" event={"ID":"4852b307-ba58-41ee-a079-ba1a7d2e3484","Type":"ContainerStarted","Data":"6aa67f7da275b84b1bc1dbcaea7457108d5ac762a7dffa4494c071a70c172a0f"} Jul 01 19:25:01.466025 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:01.465992 2585 generic.go:356] "Generic (PLEG): container finished" podID="44087f53-56ca-43d7-aaf8-582ee3ba0c1c" containerID="d8d81543cb522657137bd5b42b1782050d2fe7d0ed312f4d815d28e1732bcca2" exitCode=0 Jul 01 19:25:01.466450 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:01.466076 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" event={"ID":"44087f53-56ca-43d7-aaf8-582ee3ba0c1c","Type":"ContainerDied","Data":"d8d81543cb522657137bd5b42b1782050d2fe7d0ed312f4d815d28e1732bcca2"} Jul 01 19:25:01.466450 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:01.466445 2585 scope.go:117] "RemoveContainer" containerID="d8d81543cb522657137bd5b42b1782050d2fe7d0ed312f4d815d28e1732bcca2" Jul 01 19:25:01.467939 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:01.467850 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7584cf5bc7-lbzxm" event={"ID":"4852b307-ba58-41ee-a079-ba1a7d2e3484","Type":"ContainerStarted","Data":"25cd954a7946ec78728f62c9ed18fb3c6bfb5a243eb412aa0f30c6ecf8b0660d"} Jul 01 19:25:01.501012 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:01.500962 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7584cf5bc7-lbzxm" podStartSLOduration=2.500942877 podStartE2EDuration="2.500942877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:25:01.49942681 +0000 UTC m=+109.045803445" watchObservedRunningTime="2026-07-01 19:25:01.500942877 +0000 UTC m=+109.047319511" Jul 01 19:25:01.519771 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:01.519735 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:25:01.519871 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:01.519806 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:25:01.524679 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:01.524656 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:25:02.358224 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:02.358167 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:02.430338 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:02.430302 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:02.472775 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:02.472743 2585 generic.go:356] "Generic (PLEG): container finished" podID="8f60a799-ece9-40f1-a3ff-44f5f10ef0a0" containerID="8c3dd23869130a38239be0f50fdb7aff8c0717eb66f145d634ce875a37a10346" exitCode=0 Jul 01 19:25:02.473242 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:02.472822 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-428zv" event={"ID":"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0","Type":"ContainerDied","Data":"8c3dd23869130a38239be0f50fdb7aff8c0717eb66f145d634ce875a37a10346"} Jul 01 19:25:02.473242 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:02.473236 2585 scope.go:117] "RemoveContainer" containerID="8c3dd23869130a38239be0f50fdb7aff8c0717eb66f145d634ce875a37a10346" Jul 01 19:25:02.474943 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:02.474922 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-kmnlf" event={"ID":"44087f53-56ca-43d7-aaf8-582ee3ba0c1c","Type":"ContainerStarted","Data":"e665335fd3201f809a7fc03c2678b42bb51e2e1558462cfc7fb1f7921ac813e4"} Jul 01 19:25:02.478936 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:02.478914 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:25:02.847355 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:02.847307 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:02.951596 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:02.951563 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:03.174195 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:03.174086 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fb847847c-xqccn_0c94f978-1520-4f79-91df-f1f0d8048877/router/0.log" Jul 01 19:25:03.195622 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:03.195587 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z82nd_952e2ea8-73dc-4456-9ae4-29fc1e2cdb02/serve-healthcheck-canary/0.log" Jul 01 19:25:03.481087 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:03.480986 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-428zv" event={"ID":"8f60a799-ece9-40f1-a3ff-44f5f10ef0a0","Type":"ContainerStarted","Data":"eee834f3986210841a56abb7cd56d24dfbbb2e6c012f9f81f3b82e833acd36da"} Jul 01 19:25:09.246814 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.246772 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" podUID="94519212-92c3-47b1-98d5-a1dfff195e48" containerName="registry" containerID="cri-o://7155b15f270aa13ae4c86fed8847e561411d9001572ec66c99457d9c9268dfa8" gracePeriod=30 Jul 01 19:25:09.494648 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.494620 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:25:09.503628 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.503554 2585 generic.go:356] "Generic (PLEG): container finished" podID="94519212-92c3-47b1-98d5-a1dfff195e48" containerID="7155b15f270aa13ae4c86fed8847e561411d9001572ec66c99457d9c9268dfa8" exitCode=0 Jul 01 19:25:09.503628 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.503592 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" Jul 01 19:25:09.503802 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.503640 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" event={"ID":"94519212-92c3-47b1-98d5-a1dfff195e48","Type":"ContainerDied","Data":"7155b15f270aa13ae4c86fed8847e561411d9001572ec66c99457d9c9268dfa8"} Jul 01 19:25:09.503802 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.503692 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d9b4f4577-nvd9d" event={"ID":"94519212-92c3-47b1-98d5-a1dfff195e48","Type":"ContainerDied","Data":"f3dd7eacaf90c871d61c4843c216d1ecdc35ce34d05ec96240cff659341e29a9"} Jul 01 19:25:09.503802 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.503711 2585 scope.go:117] "RemoveContainer" containerID="7155b15f270aa13ae4c86fed8847e561411d9001572ec66c99457d9c9268dfa8" Jul 01 19:25:09.511940 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.511924 2585 scope.go:117] "RemoveContainer" containerID="7155b15f270aa13ae4c86fed8847e561411d9001572ec66c99457d9c9268dfa8" Jul 01 19:25:09.512281 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:25:09.512262 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7155b15f270aa13ae4c86fed8847e561411d9001572ec66c99457d9c9268dfa8\": container with ID starting with 7155b15f270aa13ae4c86fed8847e561411d9001572ec66c99457d9c9268dfa8 not found: ID does not exist" containerID="7155b15f270aa13ae4c86fed8847e561411d9001572ec66c99457d9c9268dfa8" Jul 01 19:25:09.512359 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.512295 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7155b15f270aa13ae4c86fed8847e561411d9001572ec66c99457d9c9268dfa8"} err="failed to get container status \"7155b15f270aa13ae4c86fed8847e561411d9001572ec66c99457d9c9268dfa8\": rpc error: code = NotFound desc = could not find container \"7155b15f270aa13ae4c86fed8847e561411d9001572ec66c99457d9c9268dfa8\": container with ID starting with 7155b15f270aa13ae4c86fed8847e561411d9001572ec66c99457d9c9268dfa8 not found: ID does not exist" Jul 01 19:25:09.532273 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.532248 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls\") pod \"94519212-92c3-47b1-98d5-a1dfff195e48\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " Jul 01 19:25:09.532415 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.532289 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/94519212-92c3-47b1-98d5-a1dfff195e48-image-registry-private-configuration\") pod \"94519212-92c3-47b1-98d5-a1dfff195e48\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " Jul 01 19:25:09.532415 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.532318 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-bound-sa-token\") pod \"94519212-92c3-47b1-98d5-a1dfff195e48\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " Jul 01 19:25:09.532415 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.532339 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/94519212-92c3-47b1-98d5-a1dfff195e48-installation-pull-secrets\") pod \"94519212-92c3-47b1-98d5-a1dfff195e48\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " Jul 01 19:25:09.532579 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.532561 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/94519212-92c3-47b1-98d5-a1dfff195e48-ca-trust-extracted\") pod \"94519212-92c3-47b1-98d5-a1dfff195e48\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " Jul 01 19:25:09.532635 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.532624 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94519212-92c3-47b1-98d5-a1dfff195e48-trusted-ca\") pod \"94519212-92c3-47b1-98d5-a1dfff195e48\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " Jul 01 19:25:09.532696 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.532681 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/94519212-92c3-47b1-98d5-a1dfff195e48-registry-certificates\") pod \"94519212-92c3-47b1-98d5-a1dfff195e48\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " Jul 01 19:25:09.532752 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.532736 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c57q8\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-kube-api-access-c57q8\") pod \"94519212-92c3-47b1-98d5-a1dfff195e48\" (UID: \"94519212-92c3-47b1-98d5-a1dfff195e48\") " Jul 01 19:25:09.533757 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.533698 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94519212-92c3-47b1-98d5-a1dfff195e48-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "94519212-92c3-47b1-98d5-a1dfff195e48" (UID: "94519212-92c3-47b1-98d5-a1dfff195e48"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:25:09.533890 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.533774 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94519212-92c3-47b1-98d5-a1dfff195e48-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "94519212-92c3-47b1-98d5-a1dfff195e48" (UID: "94519212-92c3-47b1-98d5-a1dfff195e48"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:25:09.535383 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.535317 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "94519212-92c3-47b1-98d5-a1dfff195e48" (UID: "94519212-92c3-47b1-98d5-a1dfff195e48"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 19:25:09.535514 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.535375 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94519212-92c3-47b1-98d5-a1dfff195e48-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "94519212-92c3-47b1-98d5-a1dfff195e48" (UID: "94519212-92c3-47b1-98d5-a1dfff195e48"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:25:09.535514 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.535473 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "94519212-92c3-47b1-98d5-a1dfff195e48" (UID: "94519212-92c3-47b1-98d5-a1dfff195e48"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 19:25:09.535621 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.535548 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94519212-92c3-47b1-98d5-a1dfff195e48-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "94519212-92c3-47b1-98d5-a1dfff195e48" (UID: "94519212-92c3-47b1-98d5-a1dfff195e48"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:25:09.535621 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.535608 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-kube-api-access-c57q8" (OuterVolumeSpecName: "kube-api-access-c57q8") pod "94519212-92c3-47b1-98d5-a1dfff195e48" (UID: "94519212-92c3-47b1-98d5-a1dfff195e48"). InnerVolumeSpecName "kube-api-access-c57q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 19:25:09.541322 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.541296 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94519212-92c3-47b1-98d5-a1dfff195e48-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "94519212-92c3-47b1-98d5-a1dfff195e48" (UID: "94519212-92c3-47b1-98d5-a1dfff195e48"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 01 19:25:09.634057 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.634020 2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-registry-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:09.634057 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.634053 2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/94519212-92c3-47b1-98d5-a1dfff195e48-image-registry-private-configuration\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:09.634057 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.634084 2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-bound-sa-token\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:09.634320 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.634094 2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/94519212-92c3-47b1-98d5-a1dfff195e48-installation-pull-secrets\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:09.634320 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.634104 2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/94519212-92c3-47b1-98d5-a1dfff195e48-ca-trust-extracted\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:09.634320 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.634112 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94519212-92c3-47b1-98d5-a1dfff195e48-trusted-ca\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:09.634320 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.634119 2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/94519212-92c3-47b1-98d5-a1dfff195e48-registry-certificates\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:09.634320 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.634127 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c57q8\" (UniqueName: \"kubernetes.io/projected/94519212-92c3-47b1-98d5-a1dfff195e48-kube-api-access-c57q8\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:09.828526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.828497 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d9b4f4577-nvd9d"] Jul 01 19:25:09.834033 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:09.834008 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d9b4f4577-nvd9d"] Jul 01 19:25:10.301414 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:10.301380 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:10.301803 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:10.301444 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:10.306034 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:10.306009 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:10.511952 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:10.511923 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:25:10.558760 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:10.558675 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dcc9bb7fd-znq6q"] Jul 01 19:25:10.710874 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:10.710839 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:10.793033 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:10.793000 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:10.960980 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:10.960906 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94519212-92c3-47b1-98d5-a1dfff195e48" path="/var/lib/kubelet/pods/94519212-92c3-47b1-98d5-a1dfff195e48/volumes" Jul 01 19:25:16.102134 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:16.102097 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:16.202610 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:16.202579 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:16.537868 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:16.537835 2585 generic.go:356] "Generic (PLEG): container finished" podID="5947ec3f-7e65-49cf-b511-44dc3b526ff2" containerID="03b35db84350a38e6d4639d38f8b47bfda13527ce084ed9435942dbe809034f2" exitCode=0 Jul 01 19:25:16.538025 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:16.537911 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" event={"ID":"5947ec3f-7e65-49cf-b511-44dc3b526ff2","Type":"ContainerDied","Data":"03b35db84350a38e6d4639d38f8b47bfda13527ce084ed9435942dbe809034f2"} Jul 01 19:25:16.538284 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:16.538271 2585 scope.go:117] "RemoveContainer" containerID="03b35db84350a38e6d4639d38f8b47bfda13527ce084ed9435942dbe809034f2" Jul 01 19:25:17.337021 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:17.336985 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:17.438173 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:17.438144 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:17.543671 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:17.543637 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-685pf" event={"ID":"5947ec3f-7e65-49cf-b511-44dc3b526ff2","Type":"ContainerStarted","Data":"4240ff1ea5c0a52b2badf4673309d0535adfcc2b7b3c2e9686c990ad33c053a6"} Jul 01 19:25:23.059229 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:23.059182 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:27.593572 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:27.593534 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:27.689918 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:27.689888 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:27.790305 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:27.790274 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:32.358609 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:32.358577 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:32.458444 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:32.458415 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:35.581952 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.581884 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-dcc9bb7fd-znq6q" podUID="adda2008-79e0-420b-a539-924ce999e7d9" containerName="console" containerID="cri-o://dc61952097755215287ee2d418f8678d6a8524cff0df57d329b97859f5a54298" gracePeriod=15 Jul 01 19:25:35.821842 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.821819 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:25:35.960015 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.959914 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-oauth-serving-cert\") pod \"adda2008-79e0-420b-a539-924ce999e7d9\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " Jul 01 19:25:35.960015 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.959973 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adda2008-79e0-420b-a539-924ce999e7d9-console-oauth-config\") pod \"adda2008-79e0-420b-a539-924ce999e7d9\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " Jul 01 19:25:35.960015 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.959996 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-trusted-ca-bundle\") pod \"adda2008-79e0-420b-a539-924ce999e7d9\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " Jul 01 19:25:35.960296 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.960107 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-console-config\") pod \"adda2008-79e0-420b-a539-924ce999e7d9\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " Jul 01 19:25:35.960296 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.960147 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adda2008-79e0-420b-a539-924ce999e7d9-console-serving-cert\") pod \"adda2008-79e0-420b-a539-924ce999e7d9\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " Jul 01 19:25:35.960296 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.960175 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-service-ca\") pod \"adda2008-79e0-420b-a539-924ce999e7d9\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " Jul 01 19:25:35.960296 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.960239 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxxnq\" (UniqueName: \"kubernetes.io/projected/adda2008-79e0-420b-a539-924ce999e7d9-kube-api-access-nxxnq\") pod \"adda2008-79e0-420b-a539-924ce999e7d9\" (UID: \"adda2008-79e0-420b-a539-924ce999e7d9\") " Jul 01 19:25:35.960496 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.960384 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "adda2008-79e0-420b-a539-924ce999e7d9" (UID: "adda2008-79e0-420b-a539-924ce999e7d9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:25:35.960565 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.960540 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "adda2008-79e0-420b-a539-924ce999e7d9" (UID: "adda2008-79e0-420b-a539-924ce999e7d9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:25:35.960626 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.960558 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-console-config" (OuterVolumeSpecName: "console-config") pod "adda2008-79e0-420b-a539-924ce999e7d9" (UID: "adda2008-79e0-420b-a539-924ce999e7d9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:25:35.960679 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.960607 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-service-ca" (OuterVolumeSpecName: "service-ca") pod "adda2008-79e0-420b-a539-924ce999e7d9" (UID: "adda2008-79e0-420b-a539-924ce999e7d9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:25:35.960732 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.960696 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-oauth-serving-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:35.960732 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.960716 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-trusted-ca-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:35.960807 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.960731 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-console-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:35.962481 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.962445 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adda2008-79e0-420b-a539-924ce999e7d9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "adda2008-79e0-420b-a539-924ce999e7d9" (UID: "adda2008-79e0-420b-a539-924ce999e7d9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:25:35.962481 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.962461 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adda2008-79e0-420b-a539-924ce999e7d9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "adda2008-79e0-420b-a539-924ce999e7d9" (UID: "adda2008-79e0-420b-a539-924ce999e7d9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:25:35.962625 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:35.962478 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adda2008-79e0-420b-a539-924ce999e7d9-kube-api-access-nxxnq" (OuterVolumeSpecName: "kube-api-access-nxxnq") pod "adda2008-79e0-420b-a539-924ce999e7d9" (UID: "adda2008-79e0-420b-a539-924ce999e7d9"). InnerVolumeSpecName "kube-api-access-nxxnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 19:25:36.061394 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:36.061351 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxxnq\" (UniqueName: \"kubernetes.io/projected/adda2008-79e0-420b-a539-924ce999e7d9-kube-api-access-nxxnq\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:36.061394 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:36.061390 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adda2008-79e0-420b-a539-924ce999e7d9-console-oauth-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:36.061394 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:36.061400 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adda2008-79e0-420b-a539-924ce999e7d9-console-serving-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:36.061394 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:36.061409 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adda2008-79e0-420b-a539-924ce999e7d9-service-ca\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:25:36.617984 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:36.617949 2585 generic.go:356] "Generic (PLEG): container finished" podID="adda2008-79e0-420b-a539-924ce999e7d9" containerID="dc61952097755215287ee2d418f8678d6a8524cff0df57d329b97859f5a54298" exitCode=0 Jul 01 19:25:36.618472 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:36.618002 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcc9bb7fd-znq6q" Jul 01 19:25:36.618472 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:36.618034 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcc9bb7fd-znq6q" event={"ID":"adda2008-79e0-420b-a539-924ce999e7d9","Type":"ContainerDied","Data":"dc61952097755215287ee2d418f8678d6a8524cff0df57d329b97859f5a54298"} Jul 01 19:25:36.618472 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:36.618096 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcc9bb7fd-znq6q" event={"ID":"adda2008-79e0-420b-a539-924ce999e7d9","Type":"ContainerDied","Data":"43bd8658205c5edd6c474fac33fda1a363ecd684e3fe68cc1b1a35e83548f580"} Jul 01 19:25:36.618472 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:36.618119 2585 scope.go:117] "RemoveContainer" containerID="dc61952097755215287ee2d418f8678d6a8524cff0df57d329b97859f5a54298" Jul 01 19:25:36.626224 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:36.626205 2585 scope.go:117] "RemoveContainer" containerID="dc61952097755215287ee2d418f8678d6a8524cff0df57d329b97859f5a54298" Jul 01 19:25:36.626471 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:25:36.626452 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc61952097755215287ee2d418f8678d6a8524cff0df57d329b97859f5a54298\": container with ID starting with dc61952097755215287ee2d418f8678d6a8524cff0df57d329b97859f5a54298 not found: ID does not exist" containerID="dc61952097755215287ee2d418f8678d6a8524cff0df57d329b97859f5a54298" Jul 01 19:25:36.626533 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:36.626482 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc61952097755215287ee2d418f8678d6a8524cff0df57d329b97859f5a54298"} err="failed to get container status \"dc61952097755215287ee2d418f8678d6a8524cff0df57d329b97859f5a54298\": rpc error: code = NotFound desc = could not find container \"dc61952097755215287ee2d418f8678d6a8524cff0df57d329b97859f5a54298\": container with ID starting with dc61952097755215287ee2d418f8678d6a8524cff0df57d329b97859f5a54298 not found: ID does not exist" Jul 01 19:25:36.638214 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:36.638190 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dcc9bb7fd-znq6q"] Jul 01 19:25:36.642857 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:36.642838 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dcc9bb7fd-znq6q"] Jul 01 19:25:36.961271 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:36.961197 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adda2008-79e0-420b-a539-924ce999e7d9" path="/var/lib/kubelet/pods/adda2008-79e0-420b-a539-924ce999e7d9/volumes" Jul 01 19:25:42.686953 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:42.686918 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:53.789575 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:53.789532 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:25:53.899166 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:25:53.899129 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:05.675432 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:05.675350 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:05.774524 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:05.774462 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:08.646222 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:08.646183 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:17.345586 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:17.345546 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:17.453205 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:17.453172 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:17.520954 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:17.520918 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:33.212541 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:33.212487 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:33.277472 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:33.277436 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:34.066235 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.066203 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d97d4c95c-pbf2k"] Jul 01 19:26:34.066484 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.066473 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adda2008-79e0-420b-a539-924ce999e7d9" containerName="console" Jul 01 19:26:34.066531 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.066486 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="adda2008-79e0-420b-a539-924ce999e7d9" containerName="console" Jul 01 19:26:34.066531 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.066502 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94519212-92c3-47b1-98d5-a1dfff195e48" containerName="registry" Jul 01 19:26:34.066531 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.066508 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="94519212-92c3-47b1-98d5-a1dfff195e48" containerName="registry" Jul 01 19:26:34.066675 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.066562 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="adda2008-79e0-420b-a539-924ce999e7d9" containerName="console" Jul 01 19:26:34.066675 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.066569 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="94519212-92c3-47b1-98d5-a1dfff195e48" containerName="registry" Jul 01 19:26:34.069240 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.069216 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.086727 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.086689 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d97d4c95c-pbf2k"] Jul 01 19:26:34.148672 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.148643 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:34.223679 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.223642 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h84zc\" (UniqueName: \"kubernetes.io/projected/a526b395-6720-4b7e-826f-8994d3c9922d-kube-api-access-h84zc\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.224051 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.223694 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-trusted-ca-bundle\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.224051 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.223718 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-console-config\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.224051 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.223750 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a526b395-6720-4b7e-826f-8994d3c9922d-console-serving-cert\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.224051 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.223766 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-service-ca\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.224051 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.223786 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-oauth-serving-cert\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.224051 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.223886 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a526b395-6720-4b7e-826f-8994d3c9922d-console-oauth-config\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.252171 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.252134 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:34.309143 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.309108 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:34.325165 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.325089 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a526b395-6720-4b7e-826f-8994d3c9922d-console-oauth-config\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.325165 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.325140 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h84zc\" (UniqueName: \"kubernetes.io/projected/a526b395-6720-4b7e-826f-8994d3c9922d-kube-api-access-h84zc\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.325347 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.325189 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-trusted-ca-bundle\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.325347 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.325224 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-console-config\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.325347 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.325276 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a526b395-6720-4b7e-826f-8994d3c9922d-console-serving-cert\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.325347 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.325297 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-service-ca\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.325347 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.325336 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-oauth-serving-cert\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.326156 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.326129 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-oauth-serving-cert\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.326156 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.326150 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-console-config\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.326343 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.326191 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-service-ca\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.326343 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.326236 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-trusted-ca-bundle\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.327788 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.327770 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a526b395-6720-4b7e-826f-8994d3c9922d-console-serving-cert\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.327906 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.327796 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a526b395-6720-4b7e-826f-8994d3c9922d-console-oauth-config\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.333756 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.333737 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h84zc\" (UniqueName: \"kubernetes.io/projected/a526b395-6720-4b7e-826f-8994d3c9922d-kube-api-access-h84zc\") pod \"console-7d97d4c95c-pbf2k\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.379727 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.379699 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:34.509311 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.509290 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d97d4c95c-pbf2k"] Jul 01 19:26:34.511592 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:26:34.511565 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda526b395_6720_4b7e_826f_8994d3c9922d.slice/crio-72b1d4c7f52db0ae743ab1753f834feb8ac1e86a7f5d5b10eb279fae09e02ce7 WatchSource:0}: Error finding container 72b1d4c7f52db0ae743ab1753f834feb8ac1e86a7f5d5b10eb279fae09e02ce7: Status 404 returned error can't find the container with id 72b1d4c7f52db0ae743ab1753f834feb8ac1e86a7f5d5b10eb279fae09e02ce7 Jul 01 19:26:34.833134 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.833101 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d97d4c95c-pbf2k" event={"ID":"a526b395-6720-4b7e-826f-8994d3c9922d","Type":"ContainerStarted","Data":"bccb0b0816daaf0a7cf3bf69ac696ec2adcdc4bff0a2e71a83fe177589041091"} Jul 01 19:26:34.833313 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.833156 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d97d4c95c-pbf2k" event={"ID":"a526b395-6720-4b7e-826f-8994d3c9922d","Type":"ContainerStarted","Data":"72b1d4c7f52db0ae743ab1753f834feb8ac1e86a7f5d5b10eb279fae09e02ce7"} Jul 01 19:26:34.850021 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:34.849961 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d97d4c95c-pbf2k" podStartSLOduration=0.849946082 podStartE2EDuration="849.946082ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:26:34.848846456 +0000 UTC m=+202.395223090" watchObservedRunningTime="2026-07-01 19:26:34.849946082 +0000 UTC m=+202.396322717" Jul 01 19:26:35.364283 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:35.364251 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:36.155381 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:36.155345 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:44.380193 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:44.380148 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:44.380679 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:44.380267 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:44.385086 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:44.385052 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:44.874427 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:44.874399 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:26:44.921303 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:44.921269 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7584cf5bc7-lbzxm"] Jul 01 19:26:45.040725 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:45.040693 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:45.141748 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:45.141652 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:45.200025 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:45.199990 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:50.007624 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:50.007585 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:50.107668 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:50.107634 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:50.290526 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:50.290494 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:50.389591 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:50.389557 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:50.489865 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:50.489835 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:26:51.331103 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.331051 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws"] Jul 01 19:26:51.334463 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.334447 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" Jul 01 19:26:51.337028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.337008 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 01 19:26:51.337528 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.337514 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5brdk\"" Jul 01 19:26:51.337970 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.337956 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 01 19:26:51.344284 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.344260 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws"] Jul 01 19:26:51.358477 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.358441 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws\" (UID: \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" Jul 01 19:26:51.358623 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.358490 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws\" (UID: \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" Jul 01 19:26:51.358623 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.358555 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x54q7\" (UniqueName: \"kubernetes.io/projected/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-kube-api-access-x54q7\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws\" (UID: \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" Jul 01 19:26:51.459821 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.459781 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws\" (UID: \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" Jul 01 19:26:51.459988 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.459832 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws\" (UID: \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" Jul 01 19:26:51.459988 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.459858 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x54q7\" (UniqueName: \"kubernetes.io/projected/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-kube-api-access-x54q7\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws\" (UID: \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" Jul 01 19:26:51.460188 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.460166 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws\" (UID: \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" Jul 01 19:26:51.460230 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.460191 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws\" (UID: \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" Jul 01 19:26:51.468033 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.468013 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x54q7\" (UniqueName: \"kubernetes.io/projected/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-kube-api-access-x54q7\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws\" (UID: \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" Jul 01 19:26:51.644034 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.643950 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" Jul 01 19:26:51.768606 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.768584 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws"] Jul 01 19:26:51.770805 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:26:51.770778 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f09e9f_5e36_4920_8438_b7d8147bc5e4.slice/crio-b69b07102989edb41ef1928bbf2e1ab5d5503adaff5eeb298065c51f5975d776 WatchSource:0}: Error finding container b69b07102989edb41ef1928bbf2e1ab5d5503adaff5eeb298065c51f5975d776: Status 404 returned error can't find the container with id b69b07102989edb41ef1928bbf2e1ab5d5503adaff5eeb298065c51f5975d776 Jul 01 19:26:51.897951 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:51.897869 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" event={"ID":"b9f09e9f-5e36-4920-8438-b7d8147bc5e4","Type":"ContainerStarted","Data":"b69b07102989edb41ef1928bbf2e1ab5d5503adaff5eeb298065c51f5975d776"} Jul 01 19:26:59.932335 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:59.932299 2585 generic.go:356] "Generic (PLEG): container finished" podID="b9f09e9f-5e36-4920-8438-b7d8147bc5e4" containerID="9deb57a83fd27ab99d127d72db68683e7429e6f4cdd288829a36f2289d3429eb" exitCode=0 Jul 01 19:26:59.932757 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:26:59.932386 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" event={"ID":"b9f09e9f-5e36-4920-8438-b7d8147bc5e4","Type":"ContainerDied","Data":"9deb57a83fd27ab99d127d72db68683e7429e6f4cdd288829a36f2289d3429eb"} Jul 01 19:27:01.000360 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:01.000325 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:01.101349 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:01.101313 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:02.946834 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:02.946798 2585 generic.go:356] "Generic (PLEG): container finished" podID="b9f09e9f-5e36-4920-8438-b7d8147bc5e4" containerID="11eec744fce9a9d79f741280f22538e6b67c2e1d3dd2af10113bbb6281bd2100" exitCode=0 Jul 01 19:27:02.947223 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:02.946842 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" event={"ID":"b9f09e9f-5e36-4920-8438-b7d8147bc5e4","Type":"ContainerDied","Data":"11eec744fce9a9d79f741280f22538e6b67c2e1d3dd2af10113bbb6281bd2100"} Jul 01 19:27:09.948364 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:09.948330 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7584cf5bc7-lbzxm" podUID="4852b307-ba58-41ee-a079-ba1a7d2e3484" containerName="console" containerID="cri-o://25cd954a7946ec78728f62c9ed18fb3c6bfb5a243eb412aa0f30c6ecf8b0660d" gracePeriod=15 Jul 01 19:27:09.979655 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:09.979628 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" event={"ID":"b9f09e9f-5e36-4920-8438-b7d8147bc5e4","Type":"ContainerStarted","Data":"1221d476b3e9b289cf7c7ba19a8ab43ee55ba029214cca94165195eaded881c7"} Jul 01 19:27:09.997192 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:09.997143 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" podStartSLOduration=2.17334519 podStartE2EDuration="18.997127707s" totalImagesPullingTime="16.823782517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:26:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:27:09.99676448 +0000 UTC m=+237.543141114" watchObservedRunningTime="2026-07-01 19:27:09.997127707 +0000 UTC m=+237.543504341" Jul 01 19:27:10.185747 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.185725 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:27:10.206682 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.206657 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-service-ca\") pod \"4852b307-ba58-41ee-a079-ba1a7d2e3484\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " Jul 01 19:27:10.206842 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.206730 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-serving-cert\") pod \"4852b307-ba58-41ee-a079-ba1a7d2e3484\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " Jul 01 19:27:10.206842 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.206763 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-config\") pod \"4852b307-ba58-41ee-a079-ba1a7d2e3484\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " Jul 01 19:27:10.206842 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.206805 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-trusted-ca-bundle\") pod \"4852b307-ba58-41ee-a079-ba1a7d2e3484\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " Jul 01 19:27:10.206842 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.206827 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-oauth-serving-cert\") pod \"4852b307-ba58-41ee-a079-ba1a7d2e3484\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " Jul 01 19:27:10.207048 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.206881 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tntw9\" (UniqueName: \"kubernetes.io/projected/4852b307-ba58-41ee-a079-ba1a7d2e3484-kube-api-access-tntw9\") pod \"4852b307-ba58-41ee-a079-ba1a7d2e3484\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " Jul 01 19:27:10.207048 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.206906 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-oauth-config\") pod \"4852b307-ba58-41ee-a079-ba1a7d2e3484\" (UID: \"4852b307-ba58-41ee-a079-ba1a7d2e3484\") " Jul 01 19:27:10.207595 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.207548 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-config" (OuterVolumeSpecName: "console-config") pod "4852b307-ba58-41ee-a079-ba1a7d2e3484" (UID: "4852b307-ba58-41ee-a079-ba1a7d2e3484"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:27:10.207595 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.207557 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-service-ca" (OuterVolumeSpecName: "service-ca") pod "4852b307-ba58-41ee-a079-ba1a7d2e3484" (UID: "4852b307-ba58-41ee-a079-ba1a7d2e3484"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:27:10.207595 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.207563 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4852b307-ba58-41ee-a079-ba1a7d2e3484" (UID: "4852b307-ba58-41ee-a079-ba1a7d2e3484"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:27:10.207827 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.207594 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4852b307-ba58-41ee-a079-ba1a7d2e3484" (UID: "4852b307-ba58-41ee-a079-ba1a7d2e3484"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:27:10.209330 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.209305 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4852b307-ba58-41ee-a079-ba1a7d2e3484" (UID: "4852b307-ba58-41ee-a079-ba1a7d2e3484"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:27:10.209480 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.209460 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4852b307-ba58-41ee-a079-ba1a7d2e3484-kube-api-access-tntw9" (OuterVolumeSpecName: "kube-api-access-tntw9") pod "4852b307-ba58-41ee-a079-ba1a7d2e3484" (UID: "4852b307-ba58-41ee-a079-ba1a7d2e3484"). InnerVolumeSpecName "kube-api-access-tntw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 19:27:10.209540 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.209491 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4852b307-ba58-41ee-a079-ba1a7d2e3484" (UID: "4852b307-ba58-41ee-a079-ba1a7d2e3484"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:27:10.307646 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.307602 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-serving-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:27:10.307646 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.307639 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:27:10.307646 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.307649 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-trusted-ca-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:27:10.307646 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.307659 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-oauth-serving-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:27:10.307898 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.307668 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tntw9\" (UniqueName: \"kubernetes.io/projected/4852b307-ba58-41ee-a079-ba1a7d2e3484-kube-api-access-tntw9\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:27:10.307898 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.307676 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4852b307-ba58-41ee-a079-ba1a7d2e3484-console-oauth-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:27:10.307898 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.307685 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4852b307-ba58-41ee-a079-ba1a7d2e3484-service-ca\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:27:10.985762 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.985731 2585 generic.go:356] "Generic (PLEG): container finished" podID="b9f09e9f-5e36-4920-8438-b7d8147bc5e4" containerID="1221d476b3e9b289cf7c7ba19a8ab43ee55ba029214cca94165195eaded881c7" exitCode=0 Jul 01 19:27:10.986161 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.985814 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" event={"ID":"b9f09e9f-5e36-4920-8438-b7d8147bc5e4","Type":"ContainerDied","Data":"1221d476b3e9b289cf7c7ba19a8ab43ee55ba029214cca94165195eaded881c7"} Jul 01 19:27:10.987059 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.987039 2585 generic.go:356] "Generic (PLEG): container finished" podID="4852b307-ba58-41ee-a079-ba1a7d2e3484" containerID="25cd954a7946ec78728f62c9ed18fb3c6bfb5a243eb412aa0f30c6ecf8b0660d" exitCode=0 Jul 01 19:27:10.987168 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.987093 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7584cf5bc7-lbzxm" event={"ID":"4852b307-ba58-41ee-a079-ba1a7d2e3484","Type":"ContainerDied","Data":"25cd954a7946ec78728f62c9ed18fb3c6bfb5a243eb412aa0f30c6ecf8b0660d"} Jul 01 19:27:10.987168 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.987138 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7584cf5bc7-lbzxm" event={"ID":"4852b307-ba58-41ee-a079-ba1a7d2e3484","Type":"ContainerDied","Data":"6aa67f7da275b84b1bc1dbcaea7457108d5ac762a7dffa4494c071a70c172a0f"} Jul 01 19:27:10.987168 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.987160 2585 scope.go:117] "RemoveContainer" containerID="25cd954a7946ec78728f62c9ed18fb3c6bfb5a243eb412aa0f30c6ecf8b0660d" Jul 01 19:27:10.987316 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.987101 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7584cf5bc7-lbzxm" Jul 01 19:27:10.995223 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.995200 2585 scope.go:117] "RemoveContainer" containerID="25cd954a7946ec78728f62c9ed18fb3c6bfb5a243eb412aa0f30c6ecf8b0660d" Jul 01 19:27:10.995478 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:27:10.995460 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25cd954a7946ec78728f62c9ed18fb3c6bfb5a243eb412aa0f30c6ecf8b0660d\": container with ID starting with 25cd954a7946ec78728f62c9ed18fb3c6bfb5a243eb412aa0f30c6ecf8b0660d not found: ID does not exist" containerID="25cd954a7946ec78728f62c9ed18fb3c6bfb5a243eb412aa0f30c6ecf8b0660d" Jul 01 19:27:10.995531 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:10.995487 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25cd954a7946ec78728f62c9ed18fb3c6bfb5a243eb412aa0f30c6ecf8b0660d"} err="failed to get container status \"25cd954a7946ec78728f62c9ed18fb3c6bfb5a243eb412aa0f30c6ecf8b0660d\": rpc error: code = NotFound desc = could not find container \"25cd954a7946ec78728f62c9ed18fb3c6bfb5a243eb412aa0f30c6ecf8b0660d\": container with ID starting with 25cd954a7946ec78728f62c9ed18fb3c6bfb5a243eb412aa0f30c6ecf8b0660d not found: ID does not exist" Jul 01 19:27:11.015970 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:11.015940 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7584cf5bc7-lbzxm"] Jul 01 19:27:11.019870 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:11.019845 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7584cf5bc7-lbzxm"] Jul 01 19:27:12.114981 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:12.114949 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" Jul 01 19:27:12.223661 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:12.223628 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-bundle\") pod \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\" (UID: \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\") " Jul 01 19:27:12.223823 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:12.223698 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x54q7\" (UniqueName: \"kubernetes.io/projected/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-kube-api-access-x54q7\") pod \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\" (UID: \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\") " Jul 01 19:27:12.223823 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:12.223725 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-util\") pod \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\" (UID: \"b9f09e9f-5e36-4920-8438-b7d8147bc5e4\") " Jul 01 19:27:12.224441 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:12.224413 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-bundle" (OuterVolumeSpecName: "bundle") pod "b9f09e9f-5e36-4920-8438-b7d8147bc5e4" (UID: "b9f09e9f-5e36-4920-8438-b7d8147bc5e4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 01 19:27:12.225863 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:12.225833 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-kube-api-access-x54q7" (OuterVolumeSpecName: "kube-api-access-x54q7") pod "b9f09e9f-5e36-4920-8438-b7d8147bc5e4" (UID: "b9f09e9f-5e36-4920-8438-b7d8147bc5e4"). InnerVolumeSpecName "kube-api-access-x54q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 19:27:12.228155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:12.228134 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-util" (OuterVolumeSpecName: "util") pod "b9f09e9f-5e36-4920-8438-b7d8147bc5e4" (UID: "b9f09e9f-5e36-4920-8438-b7d8147bc5e4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 01 19:27:12.324546 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:12.324512 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x54q7\" (UniqueName: \"kubernetes.io/projected/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-kube-api-access-x54q7\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:27:12.324546 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:12.324541 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-util\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:27:12.324546 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:12.324551 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9f09e9f-5e36-4920-8438-b7d8147bc5e4-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:27:12.961232 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:12.961205 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4852b307-ba58-41ee-a079-ba1a7d2e3484" path="/var/lib/kubelet/pods/4852b307-ba58-41ee-a079-ba1a7d2e3484/volumes" Jul 01 19:27:12.996325 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:12.996290 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" event={"ID":"b9f09e9f-5e36-4920-8438-b7d8147bc5e4","Type":"ContainerDied","Data":"b69b07102989edb41ef1928bbf2e1ab5d5503adaff5eeb298065c51f5975d776"} Jul 01 19:27:12.996325 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:12.996307 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5892ksws" Jul 01 19:27:12.996603 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:12.996335 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b69b07102989edb41ef1928bbf2e1ab5d5503adaff5eeb298065c51f5975d776" Jul 01 19:27:17.656090 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:17.656034 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:17.762226 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:17.762165 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:19.954658 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.954624 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5"] Jul 01 19:27:19.955055 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.954887 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4852b307-ba58-41ee-a079-ba1a7d2e3484" containerName="console" Jul 01 19:27:19.955055 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.954898 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4852b307-ba58-41ee-a079-ba1a7d2e3484" containerName="console" Jul 01 19:27:19.955055 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.954912 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f09e9f-5e36-4920-8438-b7d8147bc5e4" containerName="util" Jul 01 19:27:19.955055 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.954919 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f09e9f-5e36-4920-8438-b7d8147bc5e4" containerName="util" Jul 01 19:27:19.955055 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.954935 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f09e9f-5e36-4920-8438-b7d8147bc5e4" containerName="pull" Jul 01 19:27:19.955055 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.954940 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f09e9f-5e36-4920-8438-b7d8147bc5e4" containerName="pull" Jul 01 19:27:19.955055 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.954945 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f09e9f-5e36-4920-8438-b7d8147bc5e4" containerName="extract" Jul 01 19:27:19.955055 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.954950 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f09e9f-5e36-4920-8438-b7d8147bc5e4" containerName="extract" Jul 01 19:27:19.955055 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.954998 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f09e9f-5e36-4920-8438-b7d8147bc5e4" containerName="extract" Jul 01 19:27:19.955055 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.955006 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4852b307-ba58-41ee-a079-ba1a7d2e3484" containerName="console" Jul 01 19:27:19.958238 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.958218 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5" Jul 01 19:27:19.961521 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.961498 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Jul 01 19:27:19.961521 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.961499 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Jul 01 19:27:19.961740 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.961687 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Jul 01 19:27:19.961740 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.961723 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-b962j\"" Jul 01 19:27:19.972197 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:19.972174 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5"] Jul 01 19:27:20.086589 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:20.086555 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/de8c14c0-c988-410e-8afe-79d8aee967f0-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5\" (UID: \"de8c14c0-c988-410e-8afe-79d8aee967f0\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5" Jul 01 19:27:20.086758 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:20.086613 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzllt\" (UniqueName: \"kubernetes.io/projected/de8c14c0-c988-410e-8afe-79d8aee967f0-kube-api-access-lzllt\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5\" (UID: \"de8c14c0-c988-410e-8afe-79d8aee967f0\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5" Jul 01 19:27:20.187274 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:20.187235 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/de8c14c0-c988-410e-8afe-79d8aee967f0-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5\" (UID: \"de8c14c0-c988-410e-8afe-79d8aee967f0\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5" Jul 01 19:27:20.187274 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:20.187277 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzllt\" (UniqueName: \"kubernetes.io/projected/de8c14c0-c988-410e-8afe-79d8aee967f0-kube-api-access-lzllt\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5\" (UID: \"de8c14c0-c988-410e-8afe-79d8aee967f0\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5" Jul 01 19:27:20.189498 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:20.189477 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/de8c14c0-c988-410e-8afe-79d8aee967f0-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5\" (UID: \"de8c14c0-c988-410e-8afe-79d8aee967f0\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5" Jul 01 19:27:20.196772 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:20.196748 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzllt\" (UniqueName: \"kubernetes.io/projected/de8c14c0-c988-410e-8afe-79d8aee967f0-kube-api-access-lzllt\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5\" (UID: \"de8c14c0-c988-410e-8afe-79d8aee967f0\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5" Jul 01 19:27:20.268561 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:20.268496 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5" Jul 01 19:27:20.402854 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:20.402829 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5"] Jul 01 19:27:20.405557 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:27:20.405529 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8c14c0_c988_410e_8afe_79d8aee967f0.slice/crio-4310e5a44f8e9a6175fa3119969bac98cc9ea2424210976d5c99c66d19e33df2 WatchSource:0}: Error finding container 4310e5a44f8e9a6175fa3119969bac98cc9ea2424210976d5c99c66d19e33df2: Status 404 returned error can't find the container with id 4310e5a44f8e9a6175fa3119969bac98cc9ea2424210976d5c99c66d19e33df2 Jul 01 19:27:21.029875 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:21.029840 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5" event={"ID":"de8c14c0-c988-410e-8afe-79d8aee967f0","Type":"ContainerStarted","Data":"4310e5a44f8e9a6175fa3119969bac98cc9ea2424210976d5c99c66d19e33df2"} Jul 01 19:27:25.610981 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:25.610946 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:25.953702 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:25.953614 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:25.959573 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:25.959546 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-pb7l8"] Jul 01 19:27:25.961788 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:25.961768 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:27:25.964538 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:25.964519 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Jul 01 19:27:25.969506 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:25.969488 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Jul 01 19:27:25.969778 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:25.969764 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-6mq5g\"" Jul 01 19:27:26.038093 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.038032 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7757ae91-79df-4f92-baee-5c70469ba304-certificates\") pod \"keda-operator-68bff59c-pb7l8\" (UID: \"7757ae91-79df-4f92-baee-5c70469ba304\") " pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:27:26.038251 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.038141 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-627zk\" (UniqueName: \"kubernetes.io/projected/7757ae91-79df-4f92-baee-5c70469ba304-kube-api-access-627zk\") pod \"keda-operator-68bff59c-pb7l8\" (UID: \"7757ae91-79df-4f92-baee-5c70469ba304\") " pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:27:26.038251 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.038189 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7757ae91-79df-4f92-baee-5c70469ba304-cabundle0\") pod \"keda-operator-68bff59c-pb7l8\" (UID: \"7757ae91-79df-4f92-baee-5c70469ba304\") " pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:27:26.041512 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.041482 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-pb7l8"] Jul 01 19:27:26.055541 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.055508 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5" event={"ID":"de8c14c0-c988-410e-8afe-79d8aee967f0","Type":"ContainerStarted","Data":"1e9ada15d80afe1b88edbbb79fe6e3ad690266e6b9caf0e0fd538e6a4e0a3f3a"} Jul 01 19:27:26.055677 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.055626 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5" Jul 01 19:27:26.088705 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.088653 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5" podStartSLOduration=2.312963285 podStartE2EDuration="7.088638049s" totalImagesPullingTime="4.775674764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:27:26.087841568 +0000 UTC m=+253.634218219" watchObservedRunningTime="2026-07-01 19:27:26.088638049 +0000 UTC m=+253.635014682" Jul 01 19:27:26.132085 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.132031 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:26.139232 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.139191 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7757ae91-79df-4f92-baee-5c70469ba304-certificates\") pod \"keda-operator-68bff59c-pb7l8\" (UID: \"7757ae91-79df-4f92-baee-5c70469ba304\") " pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:27:26.139383 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.139244 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-627zk\" (UniqueName: \"kubernetes.io/projected/7757ae91-79df-4f92-baee-5c70469ba304-kube-api-access-627zk\") pod \"keda-operator-68bff59c-pb7l8\" (UID: \"7757ae91-79df-4f92-baee-5c70469ba304\") " pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:27:26.139383 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.139280 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7757ae91-79df-4f92-baee-5c70469ba304-cabundle0\") pod \"keda-operator-68bff59c-pb7l8\" (UID: \"7757ae91-79df-4f92-baee-5c70469ba304\") " pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:27:26.139508 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:27:26.139399 2585 secret.go:281] references non-existent secret key: ca.crt Jul 01 19:27:26.139508 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:27:26.139414 2585 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Jul 01 19:27:26.139508 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:27:26.139424 2585 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-pb7l8: references non-existent secret key: ca.crt Jul 01 19:27:26.139508 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:27:26.139473 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7757ae91-79df-4f92-baee-5c70469ba304-certificates podName:7757ae91-79df-4f92-baee-5c70469ba304 nodeName:}" failed. No retries permitted until 2026-07-01 19:27:26.639455579 +0000 UTC m=+254.185832209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7757ae91-79df-4f92-baee-5c70469ba304-certificates") pod "keda-operator-68bff59c-pb7l8" (UID: "7757ae91-79df-4f92-baee-5c70469ba304") : references non-existent secret key: ca.crt Jul 01 19:27:26.139956 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.139926 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7757ae91-79df-4f92-baee-5c70469ba304-cabundle0\") pod \"keda-operator-68bff59c-pb7l8\" (UID: \"7757ae91-79df-4f92-baee-5c70469ba304\") " pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:27:26.153169 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.153140 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-627zk\" (UniqueName: \"kubernetes.io/projected/7757ae91-79df-4f92-baee-5c70469ba304-kube-api-access-627zk\") pod \"keda-operator-68bff59c-pb7l8\" (UID: \"7757ae91-79df-4f92-baee-5c70469ba304\") " pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:27:26.234905 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.234786 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:26.349407 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.349372 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:26.364816 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.364784 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-28jnt"] Jul 01 19:27:26.366914 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.366899 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" Jul 01 19:27:26.371155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.371133 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Jul 01 19:27:26.392097 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.392047 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-28jnt"] Jul 01 19:27:26.441724 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.441691 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/dc8d6939-44fc-4cfe-87df-f3d1144ab17d-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-28jnt\" (UID: \"dc8d6939-44fc-4cfe-87df-f3d1144ab17d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" Jul 01 19:27:26.441899 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.441736 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdqnm\" (UniqueName: \"kubernetes.io/projected/dc8d6939-44fc-4cfe-87df-f3d1144ab17d-kube-api-access-vdqnm\") pod \"keda-metrics-apiserver-559f75f947-28jnt\" (UID: \"dc8d6939-44fc-4cfe-87df-f3d1144ab17d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" Jul 01 19:27:26.441899 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.441773 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dc8d6939-44fc-4cfe-87df-f3d1144ab17d-certificates\") pod \"keda-metrics-apiserver-559f75f947-28jnt\" (UID: \"dc8d6939-44fc-4cfe-87df-f3d1144ab17d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" Jul 01 19:27:26.542659 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.542619 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/dc8d6939-44fc-4cfe-87df-f3d1144ab17d-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-28jnt\" (UID: \"dc8d6939-44fc-4cfe-87df-f3d1144ab17d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" Jul 01 19:27:26.542817 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.542675 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdqnm\" (UniqueName: \"kubernetes.io/projected/dc8d6939-44fc-4cfe-87df-f3d1144ab17d-kube-api-access-vdqnm\") pod \"keda-metrics-apiserver-559f75f947-28jnt\" (UID: \"dc8d6939-44fc-4cfe-87df-f3d1144ab17d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" Jul 01 19:27:26.542817 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.542717 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dc8d6939-44fc-4cfe-87df-f3d1144ab17d-certificates\") pod \"keda-metrics-apiserver-559f75f947-28jnt\" (UID: \"dc8d6939-44fc-4cfe-87df-f3d1144ab17d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" Jul 01 19:27:26.542935 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:27:26.542853 2585 secret.go:281] references non-existent secret key: tls.crt Jul 01 19:27:26.542935 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:27:26.542869 2585 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt Jul 01 19:27:26.542935 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:27:26.542891 2585 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-28jnt: references non-existent secret key: tls.crt Jul 01 19:27:26.543025 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:27:26.542973 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc8d6939-44fc-4cfe-87df-f3d1144ab17d-certificates podName:dc8d6939-44fc-4cfe-87df-f3d1144ab17d nodeName:}" failed. No retries permitted until 2026-07-01 19:27:27.042952252 +0000 UTC m=+254.589328882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/dc8d6939-44fc-4cfe-87df-f3d1144ab17d-certificates") pod "keda-metrics-apiserver-559f75f947-28jnt" (UID: "dc8d6939-44fc-4cfe-87df-f3d1144ab17d") : references non-existent secret key: tls.crt Jul 01 19:27:26.543147 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.543127 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/dc8d6939-44fc-4cfe-87df-f3d1144ab17d-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-28jnt\" (UID: \"dc8d6939-44fc-4cfe-87df-f3d1144ab17d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" Jul 01 19:27:26.577342 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.577305 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdqnm\" (UniqueName: \"kubernetes.io/projected/dc8d6939-44fc-4cfe-87df-f3d1144ab17d-kube-api-access-vdqnm\") pod \"keda-metrics-apiserver-559f75f947-28jnt\" (UID: \"dc8d6939-44fc-4cfe-87df-f3d1144ab17d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" Jul 01 19:27:26.643383 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.643349 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7757ae91-79df-4f92-baee-5c70469ba304-certificates\") pod \"keda-operator-68bff59c-pb7l8\" (UID: \"7757ae91-79df-4f92-baee-5c70469ba304\") " pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:27:26.643768 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:27:26.643492 2585 secret.go:281] references non-existent secret key: ca.crt Jul 01 19:27:26.643768 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:27:26.643508 2585 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Jul 01 19:27:26.643768 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:27:26.643516 2585 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-pb7l8: references non-existent secret key: ca.crt Jul 01 19:27:26.643768 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:27:26.643569 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7757ae91-79df-4f92-baee-5c70469ba304-certificates podName:7757ae91-79df-4f92-baee-5c70469ba304 nodeName:}" failed. No retries permitted until 2026-07-01 19:27:27.643552937 +0000 UTC m=+255.189929556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7757ae91-79df-4f92-baee-5c70469ba304-certificates") pod "keda-operator-68bff59c-pb7l8" (UID: "7757ae91-79df-4f92-baee-5c70469ba304") : references non-existent secret key: ca.crt Jul 01 19:27:26.873021 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.872910 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:26.892528 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.892494 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-jwg86"] Jul 01 19:27:26.895058 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.895043 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-jwg86" Jul 01 19:27:26.897528 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.897509 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Jul 01 19:27:26.904853 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.904828 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-jwg86"] Jul 01 19:27:26.945964 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.945932 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/802899a9-7ac4-4205-a616-afe520b1f22d-certificates\") pod \"keda-admission-5fbd5c4644-jwg86\" (UID: \"802899a9-7ac4-4205-a616-afe520b1f22d\") " pod="openshift-keda/keda-admission-5fbd5c4644-jwg86" Jul 01 19:27:26.946155 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.946010 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6h7g\" (UniqueName: \"kubernetes.io/projected/802899a9-7ac4-4205-a616-afe520b1f22d-kube-api-access-m6h7g\") pod \"keda-admission-5fbd5c4644-jwg86\" (UID: \"802899a9-7ac4-4205-a616-afe520b1f22d\") " pod="openshift-keda/keda-admission-5fbd5c4644-jwg86" Jul 01 19:27:26.972419 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:26.972383 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:27.047281 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.047249 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/802899a9-7ac4-4205-a616-afe520b1f22d-certificates\") pod \"keda-admission-5fbd5c4644-jwg86\" (UID: \"802899a9-7ac4-4205-a616-afe520b1f22d\") " pod="openshift-keda/keda-admission-5fbd5c4644-jwg86" Jul 01 19:27:27.047447 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.047310 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dc8d6939-44fc-4cfe-87df-f3d1144ab17d-certificates\") pod \"keda-metrics-apiserver-559f75f947-28jnt\" (UID: \"dc8d6939-44fc-4cfe-87df-f3d1144ab17d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" Jul 01 19:27:27.047447 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.047332 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6h7g\" (UniqueName: \"kubernetes.io/projected/802899a9-7ac4-4205-a616-afe520b1f22d-kube-api-access-m6h7g\") pod \"keda-admission-5fbd5c4644-jwg86\" (UID: \"802899a9-7ac4-4205-a616-afe520b1f22d\") " pod="openshift-keda/keda-admission-5fbd5c4644-jwg86" Jul 01 19:27:27.049899 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.049873 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/802899a9-7ac4-4205-a616-afe520b1f22d-certificates\") pod \"keda-admission-5fbd5c4644-jwg86\" (UID: \"802899a9-7ac4-4205-a616-afe520b1f22d\") " pod="openshift-keda/keda-admission-5fbd5c4644-jwg86" Jul 01 19:27:27.050014 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.049921 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dc8d6939-44fc-4cfe-87df-f3d1144ab17d-certificates\") pod \"keda-metrics-apiserver-559f75f947-28jnt\" (UID: \"dc8d6939-44fc-4cfe-87df-f3d1144ab17d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" Jul 01 19:27:27.058937 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.058911 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6h7g\" (UniqueName: \"kubernetes.io/projected/802899a9-7ac4-4205-a616-afe520b1f22d-kube-api-access-m6h7g\") pod \"keda-admission-5fbd5c4644-jwg86\" (UID: \"802899a9-7ac4-4205-a616-afe520b1f22d\") " pod="openshift-keda/keda-admission-5fbd5c4644-jwg86" Jul 01 19:27:27.075003 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.074982 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:27.205826 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.205743 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-jwg86" Jul 01 19:27:27.277152 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.277123 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" Jul 01 19:27:27.345525 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.345496 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-jwg86"] Jul 01 19:27:27.347417 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:27:27.347390 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802899a9_7ac4_4205_a616_afe520b1f22d.slice/crio-c2edafd9c21673a83dfac399310c0268aaba59622e9a55579fd2993881f84b7e WatchSource:0}: Error finding container c2edafd9c21673a83dfac399310c0268aaba59622e9a55579fd2993881f84b7e: Status 404 returned error can't find the container with id c2edafd9c21673a83dfac399310c0268aaba59622e9a55579fd2993881f84b7e Jul 01 19:27:27.424991 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.424962 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:27.425688 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.425664 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-28jnt"] Jul 01 19:27:27.426220 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:27:27.426201 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8d6939_44fc_4cfe_87df_f3d1144ab17d.slice/crio-b76f7b3393e584a00439a18f06f26eed715acbe0b1c6dae0e9fb8a576bd4ff3c WatchSource:0}: Error finding container b76f7b3393e584a00439a18f06f26eed715acbe0b1c6dae0e9fb8a576bd4ff3c: Status 404 returned error can't find the container with id b76f7b3393e584a00439a18f06f26eed715acbe0b1c6dae0e9fb8a576bd4ff3c Jul 01 19:27:27.611425 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.611397 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:27.652451 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.652420 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7757ae91-79df-4f92-baee-5c70469ba304-certificates\") pod \"keda-operator-68bff59c-pb7l8\" (UID: \"7757ae91-79df-4f92-baee-5c70469ba304\") " pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:27:27.654953 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.654927 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7757ae91-79df-4f92-baee-5c70469ba304-certificates\") pod \"keda-operator-68bff59c-pb7l8\" (UID: \"7757ae91-79df-4f92-baee-5c70469ba304\") " pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:27:27.771251 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.771216 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:27:27.816317 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.816234 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:27.919875 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.919852 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-pb7l8"] Jul 01 19:27:27.921553 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:27:27.921529 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7757ae91_79df_4f92_baee_5c70469ba304.slice/crio-c8a90f03036bf36d60a132437029d79c0b8f5acddab881e7795df1443dcd5ac9 WatchSource:0}: Error finding container c8a90f03036bf36d60a132437029d79c0b8f5acddab881e7795df1443dcd5ac9: Status 404 returned error can't find the container with id c8a90f03036bf36d60a132437029d79c0b8f5acddab881e7795df1443dcd5ac9 Jul 01 19:27:27.965932 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:27.965898 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:27:28.066438 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:28.066405 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-jwg86" event={"ID":"802899a9-7ac4-4205-a616-afe520b1f22d","Type":"ContainerStarted","Data":"c2edafd9c21673a83dfac399310c0268aaba59622e9a55579fd2993881f84b7e"} Jul 01 19:27:28.067527 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:28.067503 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-pb7l8" event={"ID":"7757ae91-79df-4f92-baee-5c70469ba304","Type":"ContainerStarted","Data":"c8a90f03036bf36d60a132437029d79c0b8f5acddab881e7795df1443dcd5ac9"} Jul 01 19:27:28.068449 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:28.068420 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" event={"ID":"dc8d6939-44fc-4cfe-87df-f3d1144ab17d","Type":"ContainerStarted","Data":"b76f7b3393e584a00439a18f06f26eed715acbe0b1c6dae0e9fb8a576bd4ff3c"} Jul 01 19:27:30.078788 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:30.078746 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-jwg86" event={"ID":"802899a9-7ac4-4205-a616-afe520b1f22d","Type":"ContainerStarted","Data":"3f52441ba9191f5fa2e959a13011a2060470cf8adca9a4daa06caa84d00802b3"} Jul 01 19:27:30.079252 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:30.078821 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-jwg86" Jul 01 19:27:30.098196 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:30.098148 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-jwg86" podStartSLOduration=2.089304305 podStartE2EDuration="4.098131161s" totalImagesPullingTime="2.008826856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:27:30.096113296 +0000 UTC m=+257.642489930" watchObservedRunningTime="2026-07-01 19:27:30.098131161 +0000 UTC m=+257.644507795" Jul 01 19:27:32.095252 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:32.095209 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-pb7l8" event={"ID":"7757ae91-79df-4f92-baee-5c70469ba304","Type":"ContainerStarted","Data":"0f56e3d1f3d99122020e5db5dd4e13beda2b83490f51c7f28a705f0ad213375a"} Jul 01 19:27:32.095716 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:32.095315 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:27:32.112266 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:32.112169 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-pb7l8" podStartSLOduration=3.1743547850000002 podStartE2EDuration="7.11215355s" totalImagesPullingTime="3.937798765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:27:32.11129623 +0000 UTC m=+259.657672865" watchObservedRunningTime="2026-07-01 19:27:32.11215355 +0000 UTC m=+259.658530183" Jul 01 19:27:38.121887 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:38.121847 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" event={"ID":"dc8d6939-44fc-4cfe-87df-f3d1144ab17d","Type":"ContainerStarted","Data":"bdd70f1d7f82561573409159e2fe4d9af803d5482bca665ccf80e223b03af685"} Jul 01 19:27:38.122281 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:38.121917 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" Jul 01 19:27:38.164014 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:38.163962 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" podStartSLOduration=2.446466764 podStartE2EDuration="12.163947463s" totalImagesPullingTime="9.717480699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:27:38.163759769 +0000 UTC m=+265.710136403" watchObservedRunningTime="2026-07-01 19:27:38.163947463 +0000 UTC m=+265.710324142" Jul 01 19:27:47.062681 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:47.062655 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tkrn5" Jul 01 19:27:49.131241 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:49.131215 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-28jnt" Jul 01 19:27:51.092143 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:51.092114 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-jwg86" Jul 01 19:27:53.101262 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:27:53.101233 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-pb7l8" Jul 01 19:28:12.863994 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:12.863969 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:28:12.864924 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:12.864900 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:28:12.875539 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:12.875519 2585 kubelet.go:1648] "Image garbage collection succeeded" Jul 01 19:28:25.671078 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:25.671021 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:28:31.703132 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.703091 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:28:31.773896 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.773860 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-m7vjp"] Jul 01 19:28:31.776741 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.776725 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" Jul 01 19:28:31.779137 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.779117 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Jul 01 19:28:31.779550 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.779536 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-pmmv2\"" Jul 01 19:28:31.779997 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.779975 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 01 19:28:31.780118 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.780100 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 01 19:28:31.797274 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.797242 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-m7vjp"] Jul 01 19:28:31.799844 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.799823 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-n5k4k"] Jul 01 19:28:31.802427 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.802408 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-n5k4k" Jul 01 19:28:31.804911 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.804890 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Jul 01 19:28:31.805057 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.805032 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-rmwvb\"" Jul 01 19:28:31.809370 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.809306 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:28:31.812466 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.812441 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-n5k4k"] Jul 01 19:28:31.876504 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.876471 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpcss\" (UniqueName: \"kubernetes.io/projected/0fae09df-5ec4-4719-a8e9-7deb4bcc56ca-kube-api-access-rpcss\") pod \"llmisvc-controller-manager-67d95fdc48-n5k4k\" (UID: \"0fae09df-5ec4-4719-a8e9-7deb4bcc56ca\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-n5k4k" Jul 01 19:28:31.876668 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.876526 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fae09df-5ec4-4719-a8e9-7deb4bcc56ca-cert\") pod \"llmisvc-controller-manager-67d95fdc48-n5k4k\" (UID: \"0fae09df-5ec4-4719-a8e9-7deb4bcc56ca\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-n5k4k" Jul 01 19:28:31.876668 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.876570 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp4kk\" (UniqueName: \"kubernetes.io/projected/0ce925dc-5754-44be-9a48-cf1eecc8284a-kube-api-access-zp4kk\") pod \"kserve-controller-manager-79d94fb575-m7vjp\" (UID: \"0ce925dc-5754-44be-9a48-cf1eecc8284a\") " pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" Jul 01 19:28:31.876668 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.876599 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ce925dc-5754-44be-9a48-cf1eecc8284a-cert\") pod \"kserve-controller-manager-79d94fb575-m7vjp\" (UID: \"0ce925dc-5754-44be-9a48-cf1eecc8284a\") " pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" Jul 01 19:28:31.887910 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.887878 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:28:31.949953 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.949916 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:28:31.977620 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.977536 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpcss\" (UniqueName: \"kubernetes.io/projected/0fae09df-5ec4-4719-a8e9-7deb4bcc56ca-kube-api-access-rpcss\") pod \"llmisvc-controller-manager-67d95fdc48-n5k4k\" (UID: \"0fae09df-5ec4-4719-a8e9-7deb4bcc56ca\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-n5k4k" Jul 01 19:28:31.977620 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.977581 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fae09df-5ec4-4719-a8e9-7deb4bcc56ca-cert\") pod \"llmisvc-controller-manager-67d95fdc48-n5k4k\" (UID: \"0fae09df-5ec4-4719-a8e9-7deb4bcc56ca\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-n5k4k" Jul 01 19:28:31.977620 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.977608 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4kk\" (UniqueName: \"kubernetes.io/projected/0ce925dc-5754-44be-9a48-cf1eecc8284a-kube-api-access-zp4kk\") pod \"kserve-controller-manager-79d94fb575-m7vjp\" (UID: \"0ce925dc-5754-44be-9a48-cf1eecc8284a\") " pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" Jul 01 19:28:31.977900 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.977641 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ce925dc-5754-44be-9a48-cf1eecc8284a-cert\") pod \"kserve-controller-manager-79d94fb575-m7vjp\" (UID: \"0ce925dc-5754-44be-9a48-cf1eecc8284a\") " pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" Jul 01 19:28:31.977900 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:28:31.977715 2585 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 01 19:28:31.977900 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:28:31.977784 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fae09df-5ec4-4719-a8e9-7deb4bcc56ca-cert podName:0fae09df-5ec4-4719-a8e9-7deb4bcc56ca nodeName:}" failed. No retries permitted until 2026-07-01 19:28:32.477761791 +0000 UTC m=+320.024138417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0fae09df-5ec4-4719-a8e9-7deb4bcc56ca-cert") pod "llmisvc-controller-manager-67d95fdc48-n5k4k" (UID: "0fae09df-5ec4-4719-a8e9-7deb4bcc56ca") : secret "llmisvc-webhook-server-cert" not found Jul 01 19:28:31.980191 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:31.980166 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ce925dc-5754-44be-9a48-cf1eecc8284a-cert\") pod \"kserve-controller-manager-79d94fb575-m7vjp\" (UID: \"0ce925dc-5754-44be-9a48-cf1eecc8284a\") " pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" Jul 01 19:28:32.038569 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:32.038540 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4kk\" (UniqueName: \"kubernetes.io/projected/0ce925dc-5754-44be-9a48-cf1eecc8284a-kube-api-access-zp4kk\") pod \"kserve-controller-manager-79d94fb575-m7vjp\" (UID: \"0ce925dc-5754-44be-9a48-cf1eecc8284a\") " pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" Jul 01 19:28:32.038854 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:32.038833 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpcss\" (UniqueName: \"kubernetes.io/projected/0fae09df-5ec4-4719-a8e9-7deb4bcc56ca-kube-api-access-rpcss\") pod \"llmisvc-controller-manager-67d95fdc48-n5k4k\" (UID: \"0fae09df-5ec4-4719-a8e9-7deb4bcc56ca\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-n5k4k" Jul 01 19:28:32.086407 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:32.086380 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" Jul 01 19:28:32.216138 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:32.215992 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-m7vjp"] Jul 01 19:28:32.218316 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:28:32.218291 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce925dc_5754_44be_9a48_cf1eecc8284a.slice/crio-c29f3320cce21b1c93b853eefc6252bf984c166545cce42f2bfda29740c2888a WatchSource:0}: Error finding container c29f3320cce21b1c93b853eefc6252bf984c166545cce42f2bfda29740c2888a: Status 404 returned error can't find the container with id c29f3320cce21b1c93b853eefc6252bf984c166545cce42f2bfda29740c2888a Jul 01 19:28:32.219612 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:32.219592 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 01 19:28:32.309867 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:32.309832 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:28:32.347419 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:32.347388 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" event={"ID":"0ce925dc-5754-44be-9a48-cf1eecc8284a","Type":"ContainerStarted","Data":"c29f3320cce21b1c93b853eefc6252bf984c166545cce42f2bfda29740c2888a"} Jul 01 19:28:32.482814 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:32.482778 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fae09df-5ec4-4719-a8e9-7deb4bcc56ca-cert\") pod \"llmisvc-controller-manager-67d95fdc48-n5k4k\" (UID: \"0fae09df-5ec4-4719-a8e9-7deb4bcc56ca\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-n5k4k" Jul 01 19:28:32.485117 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:32.485100 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fae09df-5ec4-4719-a8e9-7deb4bcc56ca-cert\") pod \"llmisvc-controller-manager-67d95fdc48-n5k4k\" (UID: \"0fae09df-5ec4-4719-a8e9-7deb4bcc56ca\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-n5k4k" Jul 01 19:28:32.716305 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:32.716211 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-n5k4k" Jul 01 19:28:32.771250 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:32.771182 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:28:32.864325 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:32.864200 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:28:32.887122 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:32.887091 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-n5k4k"] Jul 01 19:28:32.892147 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:28:32.892112 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0fae09df_5ec4_4719_a8e9_7deb4bcc56ca.slice/crio-930eeac4666ba8fb09d20dd9444623b3d51a28d6b2f3e3558d2089bc4e8739d6 WatchSource:0}: Error finding container 930eeac4666ba8fb09d20dd9444623b3d51a28d6b2f3e3558d2089bc4e8739d6: Status 404 returned error can't find the container with id 930eeac4666ba8fb09d20dd9444623b3d51a28d6b2f3e3558d2089bc4e8739d6 Jul 01 19:28:33.354303 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:33.354264 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-n5k4k" event={"ID":"0fae09df-5ec4-4719-a8e9-7deb4bcc56ca","Type":"ContainerStarted","Data":"930eeac4666ba8fb09d20dd9444623b3d51a28d6b2f3e3558d2089bc4e8739d6"} Jul 01 19:28:35.367992 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:35.367954 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" event={"ID":"0ce925dc-5754-44be-9a48-cf1eecc8284a","Type":"ContainerStarted","Data":"a4a43aac4ffe6652a75aafd03c26b8fdd437b0292e63a0f9da23b47058067295"} Jul 01 19:28:35.368421 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:35.368107 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" Jul 01 19:28:35.385088 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:35.385021 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" podStartSLOduration=1.857846027 podStartE2EDuration="4.385002662s" totalImagesPullingTime="2.527156635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:28:35.384049003 +0000 UTC m=+322.930425639" watchObservedRunningTime="2026-07-01 19:28:35.385002662 +0000 UTC m=+322.931379315" Jul 01 19:28:36.373375 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:36.373338 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-n5k4k" event={"ID":"0fae09df-5ec4-4719-a8e9-7deb4bcc56ca","Type":"ContainerStarted","Data":"09aa9c2d76de36e3b60f85430efd4939b378895afb8589551d83b21c111f7b96"} Jul 01 19:28:36.373720 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:36.373419 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-n5k4k" Jul 01 19:28:36.392360 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:28:36.392312 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-n5k4k" podStartSLOduration=2.330448064 podStartE2EDuration="5.392296709s" totalImagesPullingTime="3.061848645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:28:36.390447167 +0000 UTC m=+323.936823793" watchObservedRunningTime="2026-07-01 19:28:36.392296709 +0000 UTC m=+323.938673380" Jul 01 19:29:06.379456 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:06.379380 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" Jul 01 19:29:07.379748 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:07.379710 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-n5k4k" Jul 01 19:29:08.828461 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:08.828422 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-m7vjp"] Jul 01 19:29:08.828923 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:08.828672 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" podUID="0ce925dc-5754-44be-9a48-cf1eecc8284a" containerName="manager" containerID="cri-o://a4a43aac4ffe6652a75aafd03c26b8fdd437b0292e63a0f9da23b47058067295" gracePeriod=10 Jul 01 19:29:08.896158 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:08.896124 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-xzgv5"] Jul 01 19:29:08.898387 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:08.898356 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79d94fb575-xzgv5" Jul 01 19:29:08.950142 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:08.950110 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-xzgv5"] Jul 01 19:29:08.988411 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:08.988380 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0f673c1-84cc-490e-8c86-58a4be42ce67-cert\") pod \"kserve-controller-manager-79d94fb575-xzgv5\" (UID: \"e0f673c1-84cc-490e-8c86-58a4be42ce67\") " pod="kserve/kserve-controller-manager-79d94fb575-xzgv5" Jul 01 19:29:08.988579 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:08.988430 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffnv8\" (UniqueName: \"kubernetes.io/projected/e0f673c1-84cc-490e-8c86-58a4be42ce67-kube-api-access-ffnv8\") pod \"kserve-controller-manager-79d94fb575-xzgv5\" (UID: \"e0f673c1-84cc-490e-8c86-58a4be42ce67\") " pod="kserve/kserve-controller-manager-79d94fb575-xzgv5" Jul 01 19:29:09.077579 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.077552 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" Jul 01 19:29:09.088984 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.088932 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0f673c1-84cc-490e-8c86-58a4be42ce67-cert\") pod \"kserve-controller-manager-79d94fb575-xzgv5\" (UID: \"e0f673c1-84cc-490e-8c86-58a4be42ce67\") " pod="kserve/kserve-controller-manager-79d94fb575-xzgv5" Jul 01 19:29:09.088984 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.088970 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffnv8\" (UniqueName: \"kubernetes.io/projected/e0f673c1-84cc-490e-8c86-58a4be42ce67-kube-api-access-ffnv8\") pod \"kserve-controller-manager-79d94fb575-xzgv5\" (UID: \"e0f673c1-84cc-490e-8c86-58a4be42ce67\") " pod="kserve/kserve-controller-manager-79d94fb575-xzgv5" Jul 01 19:29:09.091102 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.091076 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0f673c1-84cc-490e-8c86-58a4be42ce67-cert\") pod \"kserve-controller-manager-79d94fb575-xzgv5\" (UID: \"e0f673c1-84cc-490e-8c86-58a4be42ce67\") " pod="kserve/kserve-controller-manager-79d94fb575-xzgv5" Jul 01 19:29:09.100546 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.100524 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffnv8\" (UniqueName: \"kubernetes.io/projected/e0f673c1-84cc-490e-8c86-58a4be42ce67-kube-api-access-ffnv8\") pod \"kserve-controller-manager-79d94fb575-xzgv5\" (UID: \"e0f673c1-84cc-490e-8c86-58a4be42ce67\") " pod="kserve/kserve-controller-manager-79d94fb575-xzgv5" Jul 01 19:29:09.190083 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.190033 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp4kk\" (UniqueName: \"kubernetes.io/projected/0ce925dc-5754-44be-9a48-cf1eecc8284a-kube-api-access-zp4kk\") pod \"0ce925dc-5754-44be-9a48-cf1eecc8284a\" (UID: \"0ce925dc-5754-44be-9a48-cf1eecc8284a\") " Jul 01 19:29:09.190236 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.190105 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ce925dc-5754-44be-9a48-cf1eecc8284a-cert\") pod \"0ce925dc-5754-44be-9a48-cf1eecc8284a\" (UID: \"0ce925dc-5754-44be-9a48-cf1eecc8284a\") " Jul 01 19:29:09.192217 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.192181 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ce925dc-5754-44be-9a48-cf1eecc8284a-cert" (OuterVolumeSpecName: "cert") pod "0ce925dc-5754-44be-9a48-cf1eecc8284a" (UID: "0ce925dc-5754-44be-9a48-cf1eecc8284a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:29:09.192345 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.192243 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce925dc-5754-44be-9a48-cf1eecc8284a-kube-api-access-zp4kk" (OuterVolumeSpecName: "kube-api-access-zp4kk") pod "0ce925dc-5754-44be-9a48-cf1eecc8284a" (UID: "0ce925dc-5754-44be-9a48-cf1eecc8284a"). InnerVolumeSpecName "kube-api-access-zp4kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 19:29:09.207423 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.207395 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79d94fb575-xzgv5" Jul 01 19:29:09.291666 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.291633 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp4kk\" (UniqueName: \"kubernetes.io/projected/0ce925dc-5754-44be-9a48-cf1eecc8284a-kube-api-access-zp4kk\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:29:09.291666 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.291665 2585 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ce925dc-5754-44be-9a48-cf1eecc8284a-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:29:09.350621 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.350598 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-xzgv5"] Jul 01 19:29:09.353173 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:29:09.353142 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f673c1_84cc_490e_8c86_58a4be42ce67.slice/crio-7932a3fcc97578cad7006ea3570ac066f2938952fb7a995b2c6d00eb1590dafe WatchSource:0}: Error finding container 7932a3fcc97578cad7006ea3570ac066f2938952fb7a995b2c6d00eb1590dafe: Status 404 returned error can't find the container with id 7932a3fcc97578cad7006ea3570ac066f2938952fb7a995b2c6d00eb1590dafe Jul 01 19:29:09.516230 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.516200 2585 generic.go:356] "Generic (PLEG): container finished" podID="0ce925dc-5754-44be-9a48-cf1eecc8284a" containerID="a4a43aac4ffe6652a75aafd03c26b8fdd437b0292e63a0f9da23b47058067295" exitCode=0 Jul 01 19:29:09.516409 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.516246 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" Jul 01 19:29:09.516409 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.516284 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" event={"ID":"0ce925dc-5754-44be-9a48-cf1eecc8284a","Type":"ContainerDied","Data":"a4a43aac4ffe6652a75aafd03c26b8fdd437b0292e63a0f9da23b47058067295"} Jul 01 19:29:09.516409 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.516342 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79d94fb575-m7vjp" event={"ID":"0ce925dc-5754-44be-9a48-cf1eecc8284a","Type":"ContainerDied","Data":"c29f3320cce21b1c93b853eefc6252bf984c166545cce42f2bfda29740c2888a"} Jul 01 19:29:09.516409 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.516373 2585 scope.go:117] "RemoveContainer" containerID="a4a43aac4ffe6652a75aafd03c26b8fdd437b0292e63a0f9da23b47058067295" Jul 01 19:29:09.517537 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.517515 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79d94fb575-xzgv5" event={"ID":"e0f673c1-84cc-490e-8c86-58a4be42ce67","Type":"ContainerStarted","Data":"7932a3fcc97578cad7006ea3570ac066f2938952fb7a995b2c6d00eb1590dafe"} Jul 01 19:29:09.524798 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.524782 2585 scope.go:117] "RemoveContainer" containerID="a4a43aac4ffe6652a75aafd03c26b8fdd437b0292e63a0f9da23b47058067295" Jul 01 19:29:09.525028 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:29:09.525013 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4a43aac4ffe6652a75aafd03c26b8fdd437b0292e63a0f9da23b47058067295\": container with ID starting with a4a43aac4ffe6652a75aafd03c26b8fdd437b0292e63a0f9da23b47058067295 not found: ID does not exist" containerID="a4a43aac4ffe6652a75aafd03c26b8fdd437b0292e63a0f9da23b47058067295" Jul 01 19:29:09.525108 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.525038 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4a43aac4ffe6652a75aafd03c26b8fdd437b0292e63a0f9da23b47058067295"} err="failed to get container status \"a4a43aac4ffe6652a75aafd03c26b8fdd437b0292e63a0f9da23b47058067295\": rpc error: code = NotFound desc = could not find container \"a4a43aac4ffe6652a75aafd03c26b8fdd437b0292e63a0f9da23b47058067295\": container with ID starting with a4a43aac4ffe6652a75aafd03c26b8fdd437b0292e63a0f9da23b47058067295 not found: ID does not exist" Jul 01 19:29:09.547887 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.547860 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-m7vjp"] Jul 01 19:29:09.554949 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:09.554926 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-m7vjp"] Jul 01 19:29:10.523555 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:10.523515 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79d94fb575-xzgv5" event={"ID":"e0f673c1-84cc-490e-8c86-58a4be42ce67","Type":"ContainerStarted","Data":"9f3c0897b24b83e79ba56f483a84772cab5c9855e416ed76d41b3c2b416a6df6"} Jul 01 19:29:10.523945 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:10.523655 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-79d94fb575-xzgv5" Jul 01 19:29:10.962058 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:10.961969 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ce925dc-5754-44be-9a48-cf1eecc8284a" path="/var/lib/kubelet/pods/0ce925dc-5754-44be-9a48-cf1eecc8284a/volumes" Jul 01 19:29:41.532252 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:41.532220 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-79d94fb575-xzgv5" Jul 01 19:29:41.556635 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:41.556584 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-79d94fb575-xzgv5" podStartSLOduration=33.192166565 podStartE2EDuration="33.556568609s" totalImagesPullingTime="364.402044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:29:10.578104826 +0000 UTC m=+358.124481459" watchObservedRunningTime="2026-07-01 19:29:41.556568609 +0000 UTC m=+389.102945258" Jul 01 19:29:42.714540 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:42.714496 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:29:42.828025 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:42.827981 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:29:42.968324 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:42.968229 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:29:43.203893 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:43.203852 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:29:43.972084 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:43.972036 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:29:44.110391 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:29:44.110352 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:30:01.098734 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:01.098697 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:30:03.728126 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.728095 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-594b45c979-x9rfm"] Jul 01 19:30:03.728499 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.728425 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce925dc-5754-44be-9a48-cf1eecc8284a" containerName="manager" Jul 01 19:30:03.728499 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.728439 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce925dc-5754-44be-9a48-cf1eecc8284a" containerName="manager" Jul 01 19:30:03.728627 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.728506 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce925dc-5754-44be-9a48-cf1eecc8284a" containerName="manager" Jul 01 19:30:03.730676 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.730657 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.734504 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.734474 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a26f54f9-d4c9-424b-8871-afc6f762c801-oauth-serving-cert\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.734670 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.734646 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw6hn\" (UniqueName: \"kubernetes.io/projected/a26f54f9-d4c9-424b-8871-afc6f762c801-kube-api-access-xw6hn\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.734785 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.734733 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a26f54f9-d4c9-424b-8871-afc6f762c801-console-config\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.734785 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.734763 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a26f54f9-d4c9-424b-8871-afc6f762c801-service-ca\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.734894 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.734807 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a26f54f9-d4c9-424b-8871-afc6f762c801-console-serving-cert\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.734894 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.734842 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26f54f9-d4c9-424b-8871-afc6f762c801-trusted-ca-bundle\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.734894 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.734882 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a26f54f9-d4c9-424b-8871-afc6f762c801-console-oauth-config\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.749081 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.749042 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-594b45c979-x9rfm"] Jul 01 19:30:03.815258 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.815209 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:30:03.835221 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.835188 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a26f54f9-d4c9-424b-8871-afc6f762c801-oauth-serving-cert\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.835377 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.835227 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw6hn\" (UniqueName: \"kubernetes.io/projected/a26f54f9-d4c9-424b-8871-afc6f762c801-kube-api-access-xw6hn\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.835377 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.835286 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a26f54f9-d4c9-424b-8871-afc6f762c801-console-config\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.835377 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.835308 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a26f54f9-d4c9-424b-8871-afc6f762c801-service-ca\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.835377 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.835341 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a26f54f9-d4c9-424b-8871-afc6f762c801-console-serving-cert\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.835587 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.835460 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26f54f9-d4c9-424b-8871-afc6f762c801-trusted-ca-bundle\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.835587 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.835518 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a26f54f9-d4c9-424b-8871-afc6f762c801-console-oauth-config\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.836104 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.836051 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a26f54f9-d4c9-424b-8871-afc6f762c801-oauth-serving-cert\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.836104 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.836089 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a26f54f9-d4c9-424b-8871-afc6f762c801-console-config\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.836296 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.836113 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a26f54f9-d4c9-424b-8871-afc6f762c801-service-ca\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.836357 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.836309 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26f54f9-d4c9-424b-8871-afc6f762c801-trusted-ca-bundle\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.837767 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.837737 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a26f54f9-d4c9-424b-8871-afc6f762c801-console-oauth-config\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.837929 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.837912 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a26f54f9-d4c9-424b-8871-afc6f762c801-console-serving-cert\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.849227 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.849204 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw6hn\" (UniqueName: \"kubernetes.io/projected/a26f54f9-d4c9-424b-8871-afc6f762c801-kube-api-access-xw6hn\") pod \"console-594b45c979-x9rfm\" (UID: \"a26f54f9-d4c9-424b-8871-afc6f762c801\") " pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:03.942036 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:03.942004 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:30:04.042168 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:04.042133 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:04.184503 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:04.184476 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-594b45c979-x9rfm"] Jul 01 19:30:04.187458 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:30:04.187430 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26f54f9_d4c9_424b_8871_afc6f762c801.slice/crio-90144f82307b16489934f5f92f6798e8445f2b32b7edc2b2842ca350ee852b44 WatchSource:0}: Error finding container 90144f82307b16489934f5f92f6798e8445f2b32b7edc2b2842ca350ee852b44: Status 404 returned error can't find the container with id 90144f82307b16489934f5f92f6798e8445f2b32b7edc2b2842ca350ee852b44 Jul 01 19:30:04.759533 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:04.759502 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-594b45c979-x9rfm" event={"ID":"a26f54f9-d4c9-424b-8871-afc6f762c801","Type":"ContainerStarted","Data":"a6f74643df600bcc2c515c5ff6d57e300ee846454742ee1cc517538ca0429035"} Jul 01 19:30:04.759895 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:04.759554 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-594b45c979-x9rfm" event={"ID":"a26f54f9-d4c9-424b-8871-afc6f762c801","Type":"ContainerStarted","Data":"90144f82307b16489934f5f92f6798e8445f2b32b7edc2b2842ca350ee852b44"} Jul 01 19:30:04.788060 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:04.788013 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-594b45c979-x9rfm" podStartSLOduration=1.787996452 podStartE2EDuration="1.787996452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:30:04.78682714 +0000 UTC m=+412.333203787" watchObservedRunningTime="2026-07-01 19:30:04.787996452 +0000 UTC m=+412.334373089" Jul 01 19:30:14.042585 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:14.042547 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:14.043207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:14.042605 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:14.047579 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:14.047555 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:14.807529 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:14.807503 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-594b45c979-x9rfm" Jul 01 19:30:14.905209 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:14.905180 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d97d4c95c-pbf2k"] Jul 01 19:30:15.145648 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:15.145571 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:30:15.299023 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:15.298978 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:30:39.925661 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:39.925619 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7d97d4c95c-pbf2k" podUID="a526b395-6720-4b7e-826f-8994d3c9922d" containerName="console" containerID="cri-o://bccb0b0816daaf0a7cf3bf69ac696ec2adcdc4bff0a2e71a83fe177589041091" gracePeriod=15 Jul 01 19:30:40.164889 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.164859 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:30:40.241544 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.241464 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a526b395-6720-4b7e-826f-8994d3c9922d-console-oauth-config\") pod \"a526b395-6720-4b7e-826f-8994d3c9922d\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " Jul 01 19:30:40.241544 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.241506 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h84zc\" (UniqueName: \"kubernetes.io/projected/a526b395-6720-4b7e-826f-8994d3c9922d-kube-api-access-h84zc\") pod \"a526b395-6720-4b7e-826f-8994d3c9922d\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " Jul 01 19:30:40.241544 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.241540 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-trusted-ca-bundle\") pod \"a526b395-6720-4b7e-826f-8994d3c9922d\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " Jul 01 19:30:40.241797 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.241577 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a526b395-6720-4b7e-826f-8994d3c9922d-console-serving-cert\") pod \"a526b395-6720-4b7e-826f-8994d3c9922d\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " Jul 01 19:30:40.241797 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.241599 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-service-ca\") pod \"a526b395-6720-4b7e-826f-8994d3c9922d\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " Jul 01 19:30:40.241797 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.241618 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-oauth-serving-cert\") pod \"a526b395-6720-4b7e-826f-8994d3c9922d\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " Jul 01 19:30:40.241797 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.241642 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-console-config\") pod \"a526b395-6720-4b7e-826f-8994d3c9922d\" (UID: \"a526b395-6720-4b7e-826f-8994d3c9922d\") " Jul 01 19:30:40.242002 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.241950 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a526b395-6720-4b7e-826f-8994d3c9922d" (UID: "a526b395-6720-4b7e-826f-8994d3c9922d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:30:40.242139 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.242113 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-service-ca" (OuterVolumeSpecName: "service-ca") pod "a526b395-6720-4b7e-826f-8994d3c9922d" (UID: "a526b395-6720-4b7e-826f-8994d3c9922d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:30:40.242236 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.242107 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a526b395-6720-4b7e-826f-8994d3c9922d" (UID: "a526b395-6720-4b7e-826f-8994d3c9922d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:30:40.242236 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.242158 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-console-config" (OuterVolumeSpecName: "console-config") pod "a526b395-6720-4b7e-826f-8994d3c9922d" (UID: "a526b395-6720-4b7e-826f-8994d3c9922d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:30:40.243691 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.243671 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a526b395-6720-4b7e-826f-8994d3c9922d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a526b395-6720-4b7e-826f-8994d3c9922d" (UID: "a526b395-6720-4b7e-826f-8994d3c9922d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:30:40.243786 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.243712 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a526b395-6720-4b7e-826f-8994d3c9922d-kube-api-access-h84zc" (OuterVolumeSpecName: "kube-api-access-h84zc") pod "a526b395-6720-4b7e-826f-8994d3c9922d" (UID: "a526b395-6720-4b7e-826f-8994d3c9922d"). InnerVolumeSpecName "kube-api-access-h84zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 19:30:40.243786 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.243757 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a526b395-6720-4b7e-826f-8994d3c9922d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a526b395-6720-4b7e-826f-8994d3c9922d" (UID: "a526b395-6720-4b7e-826f-8994d3c9922d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:30:40.342961 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.342934 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h84zc\" (UniqueName: \"kubernetes.io/projected/a526b395-6720-4b7e-826f-8994d3c9922d-kube-api-access-h84zc\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:30:40.342961 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.342958 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-trusted-ca-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:30:40.342961 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.342968 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a526b395-6720-4b7e-826f-8994d3c9922d-console-serving-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:30:40.343202 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.342977 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-service-ca\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:30:40.343202 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.342985 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-oauth-serving-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:30:40.343202 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.342994 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a526b395-6720-4b7e-826f-8994d3c9922d-console-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:30:40.343202 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.343002 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a526b395-6720-4b7e-826f-8994d3c9922d-console-oauth-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:30:40.921214 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.921175 2585 generic.go:356] "Generic (PLEG): container finished" podID="a526b395-6720-4b7e-826f-8994d3c9922d" containerID="bccb0b0816daaf0a7cf3bf69ac696ec2adcdc4bff0a2e71a83fe177589041091" exitCode=0 Jul 01 19:30:40.921420 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.921224 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d97d4c95c-pbf2k" Jul 01 19:30:40.921420 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.921263 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d97d4c95c-pbf2k" event={"ID":"a526b395-6720-4b7e-826f-8994d3c9922d","Type":"ContainerDied","Data":"bccb0b0816daaf0a7cf3bf69ac696ec2adcdc4bff0a2e71a83fe177589041091"} Jul 01 19:30:40.921420 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.921326 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d97d4c95c-pbf2k" event={"ID":"a526b395-6720-4b7e-826f-8994d3c9922d","Type":"ContainerDied","Data":"72b1d4c7f52db0ae743ab1753f834feb8ac1e86a7f5d5b10eb279fae09e02ce7"} Jul 01 19:30:40.921420 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.921354 2585 scope.go:117] "RemoveContainer" containerID="bccb0b0816daaf0a7cf3bf69ac696ec2adcdc4bff0a2e71a83fe177589041091" Jul 01 19:30:40.933785 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.933627 2585 scope.go:117] "RemoveContainer" containerID="bccb0b0816daaf0a7cf3bf69ac696ec2adcdc4bff0a2e71a83fe177589041091" Jul 01 19:30:40.934024 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:30:40.933880 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bccb0b0816daaf0a7cf3bf69ac696ec2adcdc4bff0a2e71a83fe177589041091\": container with ID starting with bccb0b0816daaf0a7cf3bf69ac696ec2adcdc4bff0a2e71a83fe177589041091 not found: ID does not exist" containerID="bccb0b0816daaf0a7cf3bf69ac696ec2adcdc4bff0a2e71a83fe177589041091" Jul 01 19:30:40.934024 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.933906 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bccb0b0816daaf0a7cf3bf69ac696ec2adcdc4bff0a2e71a83fe177589041091"} err="failed to get container status \"bccb0b0816daaf0a7cf3bf69ac696ec2adcdc4bff0a2e71a83fe177589041091\": rpc error: code = NotFound desc = could not find container \"bccb0b0816daaf0a7cf3bf69ac696ec2adcdc4bff0a2e71a83fe177589041091\": container with ID starting with bccb0b0816daaf0a7cf3bf69ac696ec2adcdc4bff0a2e71a83fe177589041091 not found: ID does not exist" Jul 01 19:30:40.953994 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.953965 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d97d4c95c-pbf2k"] Jul 01 19:30:40.961600 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:40.961577 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d97d4c95c-pbf2k"] Jul 01 19:30:42.961477 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:30:42.961449 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a526b395-6720-4b7e-826f-8994d3c9922d" path="/var/lib/kubelet/pods/a526b395-6720-4b7e-826f-8994d3c9922d/volumes" Jul 01 19:31:19.037519 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:31:19.037489 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:31:19.270906 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:31:19.270867 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:31:30.833761 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:31:30.833706 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:31:31.227390 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:31:31.227300 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:31:31.419386 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:31:31.419345 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:31:31.548859 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:31:31.548833 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:31:32.117098 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:31:32.117047 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:31:32.257494 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:31:32.257457 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:32:26.486652 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:26.486561 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:32:26.676536 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:26.676504 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:32:50.762917 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.762879 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd"] Jul 01 19:32:50.763346 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.763278 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a526b395-6720-4b7e-826f-8994d3c9922d" containerName="console" Jul 01 19:32:50.763346 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.763291 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a526b395-6720-4b7e-826f-8994d3c9922d" containerName="console" Jul 01 19:32:50.763416 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.763370 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a526b395-6720-4b7e-826f-8994d3c9922d" containerName="console" Jul 01 19:32:50.766238 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.766223 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" Jul 01 19:32:50.768569 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.768534 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-8fd9c-serving-cert\"" Jul 01 19:32:50.768720 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.768603 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-8fd9c-kube-rbac-proxy-sar-config\"" Jul 01 19:32:50.768720 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.768613 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw4nc\"" Jul 01 19:32:50.769501 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.769482 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 01 19:32:50.770384 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.770366 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:32:50.782594 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.782573 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd"] Jul 01 19:32:50.828877 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.828843 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ba7a20-6f4f-444d-8471-fd4cabf42b7a-proxy-tls\") pod \"switch-graph-8fd9c-6bb6b4fcdb-4b2kd\" (UID: \"09ba7a20-6f4f-444d-8471-fd4cabf42b7a\") " pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" Jul 01 19:32:50.828877 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.828879 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ba7a20-6f4f-444d-8471-fd4cabf42b7a-openshift-service-ca-bundle\") pod \"switch-graph-8fd9c-6bb6b4fcdb-4b2kd\" (UID: \"09ba7a20-6f4f-444d-8471-fd4cabf42b7a\") " pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" Jul 01 19:32:50.887217 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.887180 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:32:50.929347 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.929316 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ba7a20-6f4f-444d-8471-fd4cabf42b7a-proxy-tls\") pod \"switch-graph-8fd9c-6bb6b4fcdb-4b2kd\" (UID: \"09ba7a20-6f4f-444d-8471-fd4cabf42b7a\") " pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" Jul 01 19:32:50.929495 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.929353 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ba7a20-6f4f-444d-8471-fd4cabf42b7a-openshift-service-ca-bundle\") pod \"switch-graph-8fd9c-6bb6b4fcdb-4b2kd\" (UID: \"09ba7a20-6f4f-444d-8471-fd4cabf42b7a\") " pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" Jul 01 19:32:50.930094 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.930056 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ba7a20-6f4f-444d-8471-fd4cabf42b7a-openshift-service-ca-bundle\") pod \"switch-graph-8fd9c-6bb6b4fcdb-4b2kd\" (UID: \"09ba7a20-6f4f-444d-8471-fd4cabf42b7a\") " pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" Jul 01 19:32:50.931714 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:50.931688 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ba7a20-6f4f-444d-8471-fd4cabf42b7a-proxy-tls\") pod \"switch-graph-8fd9c-6bb6b4fcdb-4b2kd\" (UID: \"09ba7a20-6f4f-444d-8471-fd4cabf42b7a\") " pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" Jul 01 19:32:51.076426 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:51.076398 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" Jul 01 19:32:51.217080 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:51.217031 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd"] Jul 01 19:32:51.220132 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:32:51.220106 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ba7a20_6f4f_444d_8471_fd4cabf42b7a.slice/crio-d46ed73fbbca1fab584df578fb04e4f1f790c8c8fda419f0c643ace2ef091da7 WatchSource:0}: Error finding container d46ed73fbbca1fab584df578fb04e4f1f790c8c8fda419f0c643ace2ef091da7: Status 404 returned error can't find the container with id d46ed73fbbca1fab584df578fb04e4f1f790c8c8fda419f0c643ace2ef091da7 Jul 01 19:32:51.482524 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:51.482437 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" event={"ID":"09ba7a20-6f4f-444d-8471-fd4cabf42b7a","Type":"ContainerStarted","Data":"d46ed73fbbca1fab584df578fb04e4f1f790c8c8fda419f0c643ace2ef091da7"} Jul 01 19:32:54.498962 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:54.498927 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" event={"ID":"09ba7a20-6f4f-444d-8471-fd4cabf42b7a","Type":"ContainerStarted","Data":"3b8bcb234ba8eb2e2a54ba8eba12178af3fa61c41a12bb69a30abe43921f9712"} Jul 01 19:32:54.499384 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:54.498984 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" Jul 01 19:32:54.541976 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:32:54.536590 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" podStartSLOduration=2.00165267 podStartE2EDuration="4.536574258s" totalImagesPullingTime="2.534921588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:32:54.534343491 +0000 UTC m=+582.080720125" watchObservedRunningTime="2026-07-01 19:32:54.536574258 +0000 UTC m=+582.082950892" Jul 01 19:33:00.509612 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:00.509583 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" Jul 01 19:33:01.069639 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:01.069609 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd"] Jul 01 19:33:01.069798 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:01.069772 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" podUID="09ba7a20-6f4f-444d-8471-fd4cabf42b7a" containerName="switch-graph-8fd9c" containerID="cri-o://3b8bcb234ba8eb2e2a54ba8eba12178af3fa61c41a12bb69a30abe43921f9712" gracePeriod=30 Jul 01 19:33:01.117122 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:01.117086 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:33:01.307854 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:01.307823 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:33:01.421997 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:01.421898 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:33:01.541251 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:01.541215 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:33:01.656917 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:01.656885 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:33:01.753527 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:01.753443 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:33:05.507667 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:05.507625 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" podUID="09ba7a20-6f4f-444d-8471-fd4cabf42b7a" containerName="switch-graph-8fd9c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:33:10.507490 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:10.507452 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" podUID="09ba7a20-6f4f-444d-8471-fd4cabf42b7a" containerName="switch-graph-8fd9c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:33:12.892114 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:12.892086 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:33:12.893333 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:12.893314 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:33:15.507224 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:15.507188 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" podUID="09ba7a20-6f4f-444d-8471-fd4cabf42b7a" containerName="switch-graph-8fd9c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:33:15.507599 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:15.507299 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" Jul 01 19:33:20.507524 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:20.507485 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" podUID="09ba7a20-6f4f-444d-8471-fd4cabf42b7a" containerName="switch-graph-8fd9c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:33:25.508088 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:25.508027 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" podUID="09ba7a20-6f4f-444d-8471-fd4cabf42b7a" containerName="switch-graph-8fd9c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:33:30.507385 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:30.507348 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" podUID="09ba7a20-6f4f-444d-8471-fd4cabf42b7a" containerName="switch-graph-8fd9c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:33:31.215948 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.215926 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" Jul 01 19:33:31.372228 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.372147 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ba7a20-6f4f-444d-8471-fd4cabf42b7a-proxy-tls\") pod \"09ba7a20-6f4f-444d-8471-fd4cabf42b7a\" (UID: \"09ba7a20-6f4f-444d-8471-fd4cabf42b7a\") " Jul 01 19:33:31.372228 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.372180 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ba7a20-6f4f-444d-8471-fd4cabf42b7a-openshift-service-ca-bundle\") pod \"09ba7a20-6f4f-444d-8471-fd4cabf42b7a\" (UID: \"09ba7a20-6f4f-444d-8471-fd4cabf42b7a\") " Jul 01 19:33:31.372553 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.372531 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ba7a20-6f4f-444d-8471-fd4cabf42b7a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "09ba7a20-6f4f-444d-8471-fd4cabf42b7a" (UID: "09ba7a20-6f4f-444d-8471-fd4cabf42b7a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:33:31.374301 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.374266 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ba7a20-6f4f-444d-8471-fd4cabf42b7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09ba7a20-6f4f-444d-8471-fd4cabf42b7a" (UID: "09ba7a20-6f4f-444d-8471-fd4cabf42b7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:33:31.473436 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.473400 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ba7a20-6f4f-444d-8471-fd4cabf42b7a-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:33:31.473436 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.473430 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ba7a20-6f4f-444d-8471-fd4cabf42b7a-openshift-service-ca-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:33:31.666900 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.666817 2585 generic.go:356] "Generic (PLEG): container finished" podID="09ba7a20-6f4f-444d-8471-fd4cabf42b7a" containerID="3b8bcb234ba8eb2e2a54ba8eba12178af3fa61c41a12bb69a30abe43921f9712" exitCode=0 Jul 01 19:33:31.666900 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.666864 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" Jul 01 19:33:31.666900 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.666883 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" event={"ID":"09ba7a20-6f4f-444d-8471-fd4cabf42b7a","Type":"ContainerDied","Data":"3b8bcb234ba8eb2e2a54ba8eba12178af3fa61c41a12bb69a30abe43921f9712"} Jul 01 19:33:31.667416 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.666925 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd" event={"ID":"09ba7a20-6f4f-444d-8471-fd4cabf42b7a","Type":"ContainerDied","Data":"d46ed73fbbca1fab584df578fb04e4f1f790c8c8fda419f0c643ace2ef091da7"} Jul 01 19:33:31.667416 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.666945 2585 scope.go:117] "RemoveContainer" containerID="3b8bcb234ba8eb2e2a54ba8eba12178af3fa61c41a12bb69a30abe43921f9712" Jul 01 19:33:31.675270 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.675252 2585 scope.go:117] "RemoveContainer" containerID="3b8bcb234ba8eb2e2a54ba8eba12178af3fa61c41a12bb69a30abe43921f9712" Jul 01 19:33:31.675496 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:33:31.675478 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b8bcb234ba8eb2e2a54ba8eba12178af3fa61c41a12bb69a30abe43921f9712\": container with ID starting with 3b8bcb234ba8eb2e2a54ba8eba12178af3fa61c41a12bb69a30abe43921f9712 not found: ID does not exist" containerID="3b8bcb234ba8eb2e2a54ba8eba12178af3fa61c41a12bb69a30abe43921f9712" Jul 01 19:33:31.675544 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.675503 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8bcb234ba8eb2e2a54ba8eba12178af3fa61c41a12bb69a30abe43921f9712"} err="failed to get container status \"3b8bcb234ba8eb2e2a54ba8eba12178af3fa61c41a12bb69a30abe43921f9712\": rpc error: code = NotFound desc = could not find container \"3b8bcb234ba8eb2e2a54ba8eba12178af3fa61c41a12bb69a30abe43921f9712\": container with ID starting with 3b8bcb234ba8eb2e2a54ba8eba12178af3fa61c41a12bb69a30abe43921f9712 not found: ID does not exist" Jul 01 19:33:31.689120 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.689094 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd"] Jul 01 19:33:31.692745 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:31.692724 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8fd9c-6bb6b4fcdb-4b2kd"] Jul 01 19:33:32.961801 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:32.961765 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ba7a20-6f4f-444d-8471-fd4cabf42b7a" path="/var/lib/kubelet/pods/09ba7a20-6f4f-444d-8471-fd4cabf42b7a/volumes" Jul 01 19:33:50.809322 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.809287 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-549d845845-lqf56"] Jul 01 19:33:50.809773 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.809650 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ba7a20-6f4f-444d-8471-fd4cabf42b7a" containerName="switch-graph-8fd9c" Jul 01 19:33:50.809773 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.809663 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ba7a20-6f4f-444d-8471-fd4cabf42b7a" containerName="switch-graph-8fd9c" Jul 01 19:33:50.809773 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.809711 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ba7a20-6f4f-444d-8471-fd4cabf42b7a" containerName="switch-graph-8fd9c" Jul 01 19:33:50.812851 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.812830 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" Jul 01 19:33:50.815365 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.815334 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Jul 01 19:33:50.815467 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.815362 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 01 19:33:50.815467 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.815362 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw4nc\"" Jul 01 19:33:50.816199 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.816185 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Jul 01 19:33:50.823495 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.823477 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-549d845845-lqf56"] Jul 01 19:33:50.827412 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.827389 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ced921cf-2140-4212-b411-d494b9ca0d41-proxy-tls\") pod \"model-chainer-549d845845-lqf56\" (UID: \"ced921cf-2140-4212-b411-d494b9ca0d41\") " pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" Jul 01 19:33:50.827519 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.827430 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced921cf-2140-4212-b411-d494b9ca0d41-openshift-service-ca-bundle\") pod \"model-chainer-549d845845-lqf56\" (UID: \"ced921cf-2140-4212-b411-d494b9ca0d41\") " pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" Jul 01 19:33:50.927756 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.927727 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ced921cf-2140-4212-b411-d494b9ca0d41-proxy-tls\") pod \"model-chainer-549d845845-lqf56\" (UID: \"ced921cf-2140-4212-b411-d494b9ca0d41\") " pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" Jul 01 19:33:50.927925 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.927769 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced921cf-2140-4212-b411-d494b9ca0d41-openshift-service-ca-bundle\") pod \"model-chainer-549d845845-lqf56\" (UID: \"ced921cf-2140-4212-b411-d494b9ca0d41\") " pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" Jul 01 19:33:50.927925 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:33:50.927887 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found Jul 01 19:33:50.928000 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:33:50.927980 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced921cf-2140-4212-b411-d494b9ca0d41-proxy-tls podName:ced921cf-2140-4212-b411-d494b9ca0d41 nodeName:}" failed. No retries permitted until 2026-07-01 19:33:51.427958098 +0000 UTC m=+638.974334716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ced921cf-2140-4212-b411-d494b9ca0d41-proxy-tls") pod "model-chainer-549d845845-lqf56" (UID: "ced921cf-2140-4212-b411-d494b9ca0d41") : secret "model-chainer-serving-cert" not found Jul 01 19:33:50.928473 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.928454 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced921cf-2140-4212-b411-d494b9ca0d41-openshift-service-ca-bundle\") pod \"model-chainer-549d845845-lqf56\" (UID: \"ced921cf-2140-4212-b411-d494b9ca0d41\") " pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" Jul 01 19:33:50.992770 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:50.992733 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:33:51.431494 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:51.431460 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ced921cf-2140-4212-b411-d494b9ca0d41-proxy-tls\") pod \"model-chainer-549d845845-lqf56\" (UID: \"ced921cf-2140-4212-b411-d494b9ca0d41\") " pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" Jul 01 19:33:51.433840 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:51.433820 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ced921cf-2140-4212-b411-d494b9ca0d41-proxy-tls\") pod \"model-chainer-549d845845-lqf56\" (UID: \"ced921cf-2140-4212-b411-d494b9ca0d41\") " pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" Jul 01 19:33:51.723676 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:51.723591 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" Jul 01 19:33:51.853668 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:51.853600 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-549d845845-lqf56"] Jul 01 19:33:51.856450 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:33:51.856423 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced921cf_2140_4212_b411_d494b9ca0d41.slice/crio-d6602aa205b60694aa0f68eb5d19364314421d748ae617f4eab16ea79a41e321 WatchSource:0}: Error finding container d6602aa205b60694aa0f68eb5d19364314421d748ae617f4eab16ea79a41e321: Status 404 returned error can't find the container with id d6602aa205b60694aa0f68eb5d19364314421d748ae617f4eab16ea79a41e321 Jul 01 19:33:52.758807 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:52.758714 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" event={"ID":"ced921cf-2140-4212-b411-d494b9ca0d41","Type":"ContainerStarted","Data":"3bf20c29c960b4b0b5b9b9c585405b0d8af1b79809cc3fed738811138f4baf3b"} Jul 01 19:33:52.758807 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:52.758768 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" event={"ID":"ced921cf-2140-4212-b411-d494b9ca0d41","Type":"ContainerStarted","Data":"d6602aa205b60694aa0f68eb5d19364314421d748ae617f4eab16ea79a41e321"} Jul 01 19:33:52.758982 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:52.758823 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" Jul 01 19:33:52.777896 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:52.777850 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" podStartSLOduration=2.777835208 podStartE2EDuration="2.777835208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:33:52.775258163 +0000 UTC m=+640.321634799" watchObservedRunningTime="2026-07-01 19:33:52.777835208 +0000 UTC m=+640.324211842" Jul 01 19:33:53.137477 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:53.137443 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:33:58.769322 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:33:58.769292 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" Jul 01 19:34:00.904775 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:00.904745 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-549d845845-lqf56"] Jul 01 19:34:00.905249 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:00.904926 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" podUID="ced921cf-2140-4212-b411-d494b9ca0d41" containerName="model-chainer" containerID="cri-o://3bf20c29c960b4b0b5b9b9c585405b0d8af1b79809cc3fed738811138f4baf3b" gracePeriod=30 Jul 01 19:34:00.934456 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:00.934426 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:34:01.038219 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.038180 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:34:01.149019 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.148980 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:34:01.249828 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.249742 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:34:01.350494 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.350465 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:34:01.366189 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.366158 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5"] Jul 01 19:34:01.369585 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.369566 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" Jul 01 19:34:01.371921 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.371902 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-834d0-serving-cert\"" Jul 01 19:34:01.372442 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.372423 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-834d0-kube-rbac-proxy-sar-config\"" Jul 01 19:34:01.378996 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.378969 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5"] Jul 01 19:34:01.410615 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.410572 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/757e1922-1aea-4ea4-b1f5-193de2c2eabb-proxy-tls\") pod \"switch-graph-834d0-6d7b4cc848-b2hn5\" (UID: \"757e1922-1aea-4ea4-b1f5-193de2c2eabb\") " pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" Jul 01 19:34:01.410776 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.410633 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/757e1922-1aea-4ea4-b1f5-193de2c2eabb-openshift-service-ca-bundle\") pod \"switch-graph-834d0-6d7b4cc848-b2hn5\" (UID: \"757e1922-1aea-4ea4-b1f5-193de2c2eabb\") " pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" Jul 01 19:34:01.444840 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.444805 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:34:01.512179 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.512101 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/757e1922-1aea-4ea4-b1f5-193de2c2eabb-openshift-service-ca-bundle\") pod \"switch-graph-834d0-6d7b4cc848-b2hn5\" (UID: \"757e1922-1aea-4ea4-b1f5-193de2c2eabb\") " pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" Jul 01 19:34:01.512322 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.512195 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/757e1922-1aea-4ea4-b1f5-193de2c2eabb-proxy-tls\") pod \"switch-graph-834d0-6d7b4cc848-b2hn5\" (UID: \"757e1922-1aea-4ea4-b1f5-193de2c2eabb\") " pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" Jul 01 19:34:01.512740 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.512716 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/757e1922-1aea-4ea4-b1f5-193de2c2eabb-openshift-service-ca-bundle\") pod \"switch-graph-834d0-6d7b4cc848-b2hn5\" (UID: \"757e1922-1aea-4ea4-b1f5-193de2c2eabb\") " pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" Jul 01 19:34:01.514590 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.514570 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/757e1922-1aea-4ea4-b1f5-193de2c2eabb-proxy-tls\") pod \"switch-graph-834d0-6d7b4cc848-b2hn5\" (UID: \"757e1922-1aea-4ea4-b1f5-193de2c2eabb\") " pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" Jul 01 19:34:01.551041 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.551005 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:34:01.651474 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.651438 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:34:01.681997 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.681964 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" Jul 01 19:34:01.827080 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:01.827043 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5"] Jul 01 19:34:01.831864 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:34:01.831831 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757e1922_1aea_4ea4_b1f5_193de2c2eabb.slice/crio-845ec646e31fa117fda12fb44543a972d6464fd48de5f9bea0f6ca212dce2e76 WatchSource:0}: Error finding container 845ec646e31fa117fda12fb44543a972d6464fd48de5f9bea0f6ca212dce2e76: Status 404 returned error can't find the container with id 845ec646e31fa117fda12fb44543a972d6464fd48de5f9bea0f6ca212dce2e76 Jul 01 19:34:02.806364 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:02.806330 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" event={"ID":"757e1922-1aea-4ea4-b1f5-193de2c2eabb","Type":"ContainerStarted","Data":"9caab0d1cee2a8f93a0c1d364f509f22cc2fb9e62acb4df9adb4607173af5f2c"} Jul 01 19:34:02.806740 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:02.806408 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" event={"ID":"757e1922-1aea-4ea4-b1f5-193de2c2eabb","Type":"ContainerStarted","Data":"845ec646e31fa117fda12fb44543a972d6464fd48de5f9bea0f6ca212dce2e76"} Jul 01 19:34:02.806740 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:02.806477 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" Jul 01 19:34:02.824723 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:02.824674 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" podStartSLOduration=1.824660773 podStartE2EDuration="1.824660773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:34:02.823053174 +0000 UTC m=+650.369429808" watchObservedRunningTime="2026-07-01 19:34:02.824660773 +0000 UTC m=+650.371037407" Jul 01 19:34:03.768081 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:03.768029 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" podUID="ced921cf-2140-4212-b411-d494b9ca0d41" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:34:08.768076 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:08.768037 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" podUID="ced921cf-2140-4212-b411-d494b9ca0d41" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:34:08.816197 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:08.816169 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" Jul 01 19:34:13.376019 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:13.375980 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:34:13.767531 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:13.767445 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" podUID="ced921cf-2140-4212-b411-d494b9ca0d41" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:34:13.767683 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:13.767576 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" Jul 01 19:34:18.768076 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:18.768023 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" podUID="ced921cf-2140-4212-b411-d494b9ca0d41" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:34:23.767812 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:23.767766 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" podUID="ced921cf-2140-4212-b411-d494b9ca0d41" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:34:28.767748 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:28.767706 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" podUID="ced921cf-2140-4212-b411-d494b9ca0d41" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:34:29.815081 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:29.815033 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:34:31.562795 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.562767 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" Jul 01 19:34:31.658489 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.658457 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced921cf-2140-4212-b411-d494b9ca0d41-openshift-service-ca-bundle\") pod \"ced921cf-2140-4212-b411-d494b9ca0d41\" (UID: \"ced921cf-2140-4212-b411-d494b9ca0d41\") " Jul 01 19:34:31.658653 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.658529 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ced921cf-2140-4212-b411-d494b9ca0d41-proxy-tls\") pod \"ced921cf-2140-4212-b411-d494b9ca0d41\" (UID: \"ced921cf-2140-4212-b411-d494b9ca0d41\") " Jul 01 19:34:31.658870 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.658852 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced921cf-2140-4212-b411-d494b9ca0d41-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ced921cf-2140-4212-b411-d494b9ca0d41" (UID: "ced921cf-2140-4212-b411-d494b9ca0d41"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:34:31.660601 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.660566 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced921cf-2140-4212-b411-d494b9ca0d41-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ced921cf-2140-4212-b411-d494b9ca0d41" (UID: "ced921cf-2140-4212-b411-d494b9ca0d41"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:34:31.759205 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.759115 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ced921cf-2140-4212-b411-d494b9ca0d41-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:34:31.759205 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.759147 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced921cf-2140-4212-b411-d494b9ca0d41-openshift-service-ca-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:34:31.931723 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.931689 2585 generic.go:356] "Generic (PLEG): container finished" podID="ced921cf-2140-4212-b411-d494b9ca0d41" containerID="3bf20c29c960b4b0b5b9b9c585405b0d8af1b79809cc3fed738811138f4baf3b" exitCode=0 Jul 01 19:34:31.931893 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.931732 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" Jul 01 19:34:31.931893 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.931753 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" event={"ID":"ced921cf-2140-4212-b411-d494b9ca0d41","Type":"ContainerDied","Data":"3bf20c29c960b4b0b5b9b9c585405b0d8af1b79809cc3fed738811138f4baf3b"} Jul 01 19:34:31.931893 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.931795 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-549d845845-lqf56" event={"ID":"ced921cf-2140-4212-b411-d494b9ca0d41","Type":"ContainerDied","Data":"d6602aa205b60694aa0f68eb5d19364314421d748ae617f4eab16ea79a41e321"} Jul 01 19:34:31.931893 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.931817 2585 scope.go:117] "RemoveContainer" containerID="3bf20c29c960b4b0b5b9b9c585405b0d8af1b79809cc3fed738811138f4baf3b" Jul 01 19:34:31.942133 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.942111 2585 scope.go:117] "RemoveContainer" containerID="3bf20c29c960b4b0b5b9b9c585405b0d8af1b79809cc3fed738811138f4baf3b" Jul 01 19:34:31.942577 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:34:31.942505 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf20c29c960b4b0b5b9b9c585405b0d8af1b79809cc3fed738811138f4baf3b\": container with ID starting with 3bf20c29c960b4b0b5b9b9c585405b0d8af1b79809cc3fed738811138f4baf3b not found: ID does not exist" containerID="3bf20c29c960b4b0b5b9b9c585405b0d8af1b79809cc3fed738811138f4baf3b" Jul 01 19:34:31.942577 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.942543 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf20c29c960b4b0b5b9b9c585405b0d8af1b79809cc3fed738811138f4baf3b"} err="failed to get container status \"3bf20c29c960b4b0b5b9b9c585405b0d8af1b79809cc3fed738811138f4baf3b\": rpc error: code = NotFound desc = could not find container \"3bf20c29c960b4b0b5b9b9c585405b0d8af1b79809cc3fed738811138f4baf3b\": container with ID starting with 3bf20c29c960b4b0b5b9b9c585405b0d8af1b79809cc3fed738811138f4baf3b not found: ID does not exist" Jul 01 19:34:31.961640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.961609 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-549d845845-lqf56"] Jul 01 19:34:31.971798 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:31.971766 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-549d845845-lqf56"] Jul 01 19:34:32.962194 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:32.962152 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ced921cf-2140-4212-b411-d494b9ca0d41" path="/var/lib/kubelet/pods/ced921cf-2140-4212-b411-d494b9ca0d41/volumes" Jul 01 19:34:40.646390 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:34:40.646349 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:35:01.102225 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.102194 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs"] Jul 01 19:35:01.102675 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.102553 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced921cf-2140-4212-b411-d494b9ca0d41" containerName="model-chainer" Jul 01 19:35:01.102675 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.102564 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced921cf-2140-4212-b411-d494b9ca0d41" containerName="model-chainer" Jul 01 19:35:01.102675 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.102619 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ced921cf-2140-4212-b411-d494b9ca0d41" containerName="model-chainer" Jul 01 19:35:01.105588 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.105571 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" Jul 01 19:35:01.108012 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.107992 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-01ba2-serving-cert\"" Jul 01 19:35:01.108447 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.108433 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-01ba2-kube-rbac-proxy-sar-config\"" Jul 01 19:35:01.123760 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.123735 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs"] Jul 01 19:35:01.148284 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.148250 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:35:01.189892 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.189861 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75fe986e-063f-4101-8dee-3b49f2451d6a-proxy-tls\") pod \"sequence-graph-01ba2-68958d669b-8zsrs\" (UID: \"75fe986e-063f-4101-8dee-3b49f2451d6a\") " pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" Jul 01 19:35:01.190104 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.189954 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75fe986e-063f-4101-8dee-3b49f2451d6a-openshift-service-ca-bundle\") pod \"sequence-graph-01ba2-68958d669b-8zsrs\" (UID: \"75fe986e-063f-4101-8dee-3b49f2451d6a\") " pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" Jul 01 19:35:01.291434 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.291398 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75fe986e-063f-4101-8dee-3b49f2451d6a-openshift-service-ca-bundle\") pod \"sequence-graph-01ba2-68958d669b-8zsrs\" (UID: \"75fe986e-063f-4101-8dee-3b49f2451d6a\") " pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" Jul 01 19:35:01.291600 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.291489 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75fe986e-063f-4101-8dee-3b49f2451d6a-proxy-tls\") pod \"sequence-graph-01ba2-68958d669b-8zsrs\" (UID: \"75fe986e-063f-4101-8dee-3b49f2451d6a\") " pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" Jul 01 19:35:01.291648 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:35:01.291598 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-01ba2-serving-cert: secret "sequence-graph-01ba2-serving-cert" not found Jul 01 19:35:01.291700 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:35:01.291687 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75fe986e-063f-4101-8dee-3b49f2451d6a-proxy-tls podName:75fe986e-063f-4101-8dee-3b49f2451d6a nodeName:}" failed. No retries permitted until 2026-07-01 19:35:01.79165582 +0000 UTC m=+709.338032436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/75fe986e-063f-4101-8dee-3b49f2451d6a-proxy-tls") pod "sequence-graph-01ba2-68958d669b-8zsrs" (UID: "75fe986e-063f-4101-8dee-3b49f2451d6a") : secret "sequence-graph-01ba2-serving-cert" not found Jul 01 19:35:01.292117 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.292098 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75fe986e-063f-4101-8dee-3b49f2451d6a-openshift-service-ca-bundle\") pod \"sequence-graph-01ba2-68958d669b-8zsrs\" (UID: \"75fe986e-063f-4101-8dee-3b49f2451d6a\") " pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" Jul 01 19:35:01.418883 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.418790 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:35:01.795797 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.795756 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75fe986e-063f-4101-8dee-3b49f2451d6a-proxy-tls\") pod \"sequence-graph-01ba2-68958d669b-8zsrs\" (UID: \"75fe986e-063f-4101-8dee-3b49f2451d6a\") " pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" Jul 01 19:35:01.798270 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:01.798245 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75fe986e-063f-4101-8dee-3b49f2451d6a-proxy-tls\") pod \"sequence-graph-01ba2-68958d669b-8zsrs\" (UID: \"75fe986e-063f-4101-8dee-3b49f2451d6a\") " pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" Jul 01 19:35:02.016138 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:02.016096 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" Jul 01 19:35:02.149276 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:02.149125 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs"] Jul 01 19:35:02.151946 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:35:02.151922 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75fe986e_063f_4101_8dee_3b49f2451d6a.slice/crio-4d1850c0bd71802874d89e3c96263d0dfe3fee41b09868602bd973579df78d42 WatchSource:0}: Error finding container 4d1850c0bd71802874d89e3c96263d0dfe3fee41b09868602bd973579df78d42: Status 404 returned error can't find the container with id 4d1850c0bd71802874d89e3c96263d0dfe3fee41b09868602bd973579df78d42 Jul 01 19:35:03.070553 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:03.070514 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" event={"ID":"75fe986e-063f-4101-8dee-3b49f2451d6a","Type":"ContainerStarted","Data":"dc2a4812e0169836198c9b0678a7c3d09f12fbce1d79263edc974930bddc5eeb"} Jul 01 19:35:03.070729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:03.070566 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" event={"ID":"75fe986e-063f-4101-8dee-3b49f2451d6a","Type":"ContainerStarted","Data":"4d1850c0bd71802874d89e3c96263d0dfe3fee41b09868602bd973579df78d42"} Jul 01 19:35:03.070729 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:03.070596 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" Jul 01 19:35:03.086171 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:03.086125 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" podStartSLOduration=2.086109424 podStartE2EDuration="2.086109424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:35:03.085294235 +0000 UTC m=+710.631670870" watchObservedRunningTime="2026-07-01 19:35:03.086109424 +0000 UTC m=+710.632486057" Jul 01 19:35:09.082151 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:09.082059 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" Jul 01 19:35:56.186053 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:56.186005 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:35:56.257964 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:35:56.257928 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:37:18.498837 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:37:18.498786 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:38:12.922237 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:38:12.922164 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:38:12.925184 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:38:12.925161 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:40:45.154626 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:40:45.154594 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:42:16.034839 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:16.034806 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5"] Jul 01 19:42:16.035377 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:16.035004 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" podUID="757e1922-1aea-4ea4-b1f5-193de2c2eabb" containerName="switch-graph-834d0" containerID="cri-o://9caab0d1cee2a8f93a0c1d364f509f22cc2fb9e62acb4df9adb4607173af5f2c" gracePeriod=30 Jul 01 19:42:16.070880 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:16.070848 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:42:16.196962 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:16.196917 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:42:16.293562 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:16.293468 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:42:16.364721 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:16.364685 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:42:16.433420 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:16.433381 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:42:16.684816 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:16.684726 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:42:18.814922 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:18.814886 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" podUID="757e1922-1aea-4ea4-b1f5-193de2c2eabb" containerName="switch-graph-834d0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:42:23.814843 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:23.814795 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" podUID="757e1922-1aea-4ea4-b1f5-193de2c2eabb" containerName="switch-graph-834d0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:42:28.814878 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:28.814841 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" podUID="757e1922-1aea-4ea4-b1f5-193de2c2eabb" containerName="switch-graph-834d0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:42:28.815329 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:28.814958 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" Jul 01 19:42:33.814811 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:33.814769 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" podUID="757e1922-1aea-4ea4-b1f5-193de2c2eabb" containerName="switch-graph-834d0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:42:38.815042 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:38.814992 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" podUID="757e1922-1aea-4ea4-b1f5-193de2c2eabb" containerName="switch-graph-834d0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:42:43.814476 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:43.814432 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" podUID="757e1922-1aea-4ea4-b1f5-193de2c2eabb" containerName="switch-graph-834d0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:42:46.104349 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:46.104315 2585 generic.go:356] "Generic (PLEG): container finished" podID="757e1922-1aea-4ea4-b1f5-193de2c2eabb" containerID="9caab0d1cee2a8f93a0c1d364f509f22cc2fb9e62acb4df9adb4607173af5f2c" exitCode=0 Jul 01 19:42:46.104815 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:46.104383 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" event={"ID":"757e1922-1aea-4ea4-b1f5-193de2c2eabb","Type":"ContainerDied","Data":"9caab0d1cee2a8f93a0c1d364f509f22cc2fb9e62acb4df9adb4607173af5f2c"} Jul 01 19:42:46.187668 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:46.187642 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" Jul 01 19:42:46.288942 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:46.288856 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/757e1922-1aea-4ea4-b1f5-193de2c2eabb-proxy-tls\") pod \"757e1922-1aea-4ea4-b1f5-193de2c2eabb\" (UID: \"757e1922-1aea-4ea4-b1f5-193de2c2eabb\") " Jul 01 19:42:46.288942 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:46.288927 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/757e1922-1aea-4ea4-b1f5-193de2c2eabb-openshift-service-ca-bundle\") pod \"757e1922-1aea-4ea4-b1f5-193de2c2eabb\" (UID: \"757e1922-1aea-4ea4-b1f5-193de2c2eabb\") " Jul 01 19:42:46.289329 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:46.289297 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757e1922-1aea-4ea4-b1f5-193de2c2eabb-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "757e1922-1aea-4ea4-b1f5-193de2c2eabb" (UID: "757e1922-1aea-4ea4-b1f5-193de2c2eabb"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:42:46.290947 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:46.290922 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757e1922-1aea-4ea4-b1f5-193de2c2eabb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "757e1922-1aea-4ea4-b1f5-193de2c2eabb" (UID: "757e1922-1aea-4ea4-b1f5-193de2c2eabb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:42:46.390414 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:46.390381 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/757e1922-1aea-4ea4-b1f5-193de2c2eabb-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:42:46.390414 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:46.390412 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/757e1922-1aea-4ea4-b1f5-193de2c2eabb-openshift-service-ca-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:42:47.109851 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:47.109815 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" event={"ID":"757e1922-1aea-4ea4-b1f5-193de2c2eabb","Type":"ContainerDied","Data":"845ec646e31fa117fda12fb44543a972d6464fd48de5f9bea0f6ca212dce2e76"} Jul 01 19:42:47.110291 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:47.109874 2585 scope.go:117] "RemoveContainer" containerID="9caab0d1cee2a8f93a0c1d364f509f22cc2fb9e62acb4df9adb4607173af5f2c" Jul 01 19:42:47.110291 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:47.109830 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5" Jul 01 19:42:47.129939 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:47.129907 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5"] Jul 01 19:42:47.131664 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:47.131641 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-834d0-6d7b4cc848-b2hn5"] Jul 01 19:42:48.961559 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:42:48.961527 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="757e1922-1aea-4ea4-b1f5-193de2c2eabb" path="/var/lib/kubelet/pods/757e1922-1aea-4ea4-b1f5-193de2c2eabb/volumes" Jul 01 19:43:12.947297 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:12.947269 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:43:12.950785 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:12.950764 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:43:15.858838 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:15.858801 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs"] Jul 01 19:43:15.859300 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:15.859097 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" podUID="75fe986e-063f-4101-8dee-3b49f2451d6a" containerName="sequence-graph-01ba2" containerID="cri-o://dc2a4812e0169836198c9b0678a7c3d09f12fbce1d79263edc974930bddc5eeb" gracePeriod=30 Jul 01 19:43:15.880191 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:15.880161 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:43:15.986878 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:15.986846 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:43:16.086613 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.086582 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:43:16.197569 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.197484 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:43:16.290543 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.290506 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:43:16.320638 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.320601 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq"] Jul 01 19:43:16.320972 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.320958 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="757e1922-1aea-4ea4-b1f5-193de2c2eabb" containerName="switch-graph-834d0" Jul 01 19:43:16.321026 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.320974 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="757e1922-1aea-4ea4-b1f5-193de2c2eabb" containerName="switch-graph-834d0" Jul 01 19:43:16.321077 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.321030 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="757e1922-1aea-4ea4-b1f5-193de2c2eabb" containerName="switch-graph-834d0" Jul 01 19:43:16.325307 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.325287 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" Jul 01 19:43:16.329329 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.329302 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-ac147-kube-rbac-proxy-sar-config\"" Jul 01 19:43:16.329452 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.329355 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-ac147-serving-cert\"" Jul 01 19:43:16.335668 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.335645 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq"] Jul 01 19:43:16.392376 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.392335 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:43:16.439397 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.439364 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ba94df5-3413-40d5-834a-f447887dee1e-proxy-tls\") pod \"ensemble-graph-ac147-5f4c47599d-4q9vq\" (UID: \"1ba94df5-3413-40d5-834a-f447887dee1e\") " pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" Jul 01 19:43:16.439569 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.439417 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba94df5-3413-40d5-834a-f447887dee1e-openshift-service-ca-bundle\") pod \"ensemble-graph-ac147-5f4c47599d-4q9vq\" (UID: \"1ba94df5-3413-40d5-834a-f447887dee1e\") " pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" Jul 01 19:43:16.491812 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.491726 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:43:16.539955 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.539920 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ba94df5-3413-40d5-834a-f447887dee1e-proxy-tls\") pod \"ensemble-graph-ac147-5f4c47599d-4q9vq\" (UID: \"1ba94df5-3413-40d5-834a-f447887dee1e\") " pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" Jul 01 19:43:16.540128 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.539968 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba94df5-3413-40d5-834a-f447887dee1e-openshift-service-ca-bundle\") pod \"ensemble-graph-ac147-5f4c47599d-4q9vq\" (UID: \"1ba94df5-3413-40d5-834a-f447887dee1e\") " pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" Jul 01 19:43:16.540592 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.540572 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba94df5-3413-40d5-834a-f447887dee1e-openshift-service-ca-bundle\") pod \"ensemble-graph-ac147-5f4c47599d-4q9vq\" (UID: \"1ba94df5-3413-40d5-834a-f447887dee1e\") " pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" Jul 01 19:43:16.542423 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.542397 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ba94df5-3413-40d5-834a-f447887dee1e-proxy-tls\") pod \"ensemble-graph-ac147-5f4c47599d-4q9vq\" (UID: \"1ba94df5-3413-40d5-834a-f447887dee1e\") " pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" Jul 01 19:43:16.636899 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.636860 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" Jul 01 19:43:16.766622 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:16.766599 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq"] Jul 01 19:43:16.768883 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:43:16.768858 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba94df5_3413_40d5_834a_f447887dee1e.slice/crio-ef111cb2d74902d2bfe2e018b08607b4242360429d6edc93233a2db31234a621 WatchSource:0}: Error finding container ef111cb2d74902d2bfe2e018b08607b4242360429d6edc93233a2db31234a621: Status 404 returned error can't find the container with id ef111cb2d74902d2bfe2e018b08607b4242360429d6edc93233a2db31234a621 Jul 01 19:43:17.246603 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:17.246568 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" event={"ID":"1ba94df5-3413-40d5-834a-f447887dee1e","Type":"ContainerStarted","Data":"b51f668157e427a201c00e183d8633f8c46643ff2d47293989dd1f5ea525a760"} Jul 01 19:43:17.247113 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:17.246620 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" event={"ID":"1ba94df5-3413-40d5-834a-f447887dee1e","Type":"ContainerStarted","Data":"ef111cb2d74902d2bfe2e018b08607b4242360429d6edc93233a2db31234a621"} Jul 01 19:43:17.247113 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:17.246677 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" Jul 01 19:43:17.265123 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:17.265038 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" podStartSLOduration=1.2650233929999999 podStartE2EDuration="1.265023393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:43:17.2644912 +0000 UTC m=+1204.810867844" watchObservedRunningTime="2026-07-01 19:43:17.265023393 +0000 UTC m=+1204.811400027" Jul 01 19:43:19.079438 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:19.079403 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" podUID="75fe986e-063f-4101-8dee-3b49f2451d6a" containerName="sequence-graph-01ba2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:43:23.256265 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:23.256237 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" Jul 01 19:43:24.079369 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:24.079334 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" podUID="75fe986e-063f-4101-8dee-3b49f2451d6a" containerName="sequence-graph-01ba2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:43:26.327522 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:26.327485 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq"] Jul 01 19:43:26.327976 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:26.327753 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" podUID="1ba94df5-3413-40d5-834a-f447887dee1e" containerName="ensemble-graph-ac147" containerID="cri-o://b51f668157e427a201c00e183d8633f8c46643ff2d47293989dd1f5ea525a760" gracePeriod=30 Jul 01 19:43:26.349982 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:26.349946 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:43:26.463673 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:26.463641 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:43:26.561008 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:26.560976 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:43:26.657987 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:26.657897 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:43:26.742393 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:26.742360 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:43:26.866397 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:26.866360 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:43:27.145980 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:27.145944 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:43:28.255671 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:28.255635 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" podUID="1ba94df5-3413-40d5-834a-f447887dee1e" containerName="ensemble-graph-ac147" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:43:29.079687 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:29.079649 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" podUID="75fe986e-063f-4101-8dee-3b49f2451d6a" containerName="sequence-graph-01ba2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:43:29.079870 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:29.079753 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" Jul 01 19:43:33.254634 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:33.254596 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" podUID="1ba94df5-3413-40d5-834a-f447887dee1e" containerName="ensemble-graph-ac147" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:43:34.079327 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:34.079287 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" podUID="75fe986e-063f-4101-8dee-3b49f2451d6a" containerName="sequence-graph-01ba2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:43:38.255275 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:38.255240 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" podUID="1ba94df5-3413-40d5-834a-f447887dee1e" containerName="ensemble-graph-ac147" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:43:38.255650 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:38.255345 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" Jul 01 19:43:39.079805 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:39.079765 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" podUID="75fe986e-063f-4101-8dee-3b49f2451d6a" containerName="sequence-graph-01ba2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:43:43.255547 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:43.255510 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" podUID="1ba94df5-3413-40d5-834a-f447887dee1e" containerName="ensemble-graph-ac147" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:43:44.079097 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:44.079040 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" podUID="75fe986e-063f-4101-8dee-3b49f2451d6a" containerName="sequence-graph-01ba2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:43:46.009280 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.009255 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" Jul 01 19:43:46.077001 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.076966 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75fe986e-063f-4101-8dee-3b49f2451d6a-openshift-service-ca-bundle\") pod \"75fe986e-063f-4101-8dee-3b49f2451d6a\" (UID: \"75fe986e-063f-4101-8dee-3b49f2451d6a\") " Jul 01 19:43:46.077216 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.077015 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75fe986e-063f-4101-8dee-3b49f2451d6a-proxy-tls\") pod \"75fe986e-063f-4101-8dee-3b49f2451d6a\" (UID: \"75fe986e-063f-4101-8dee-3b49f2451d6a\") " Jul 01 19:43:46.077371 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.077347 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75fe986e-063f-4101-8dee-3b49f2451d6a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "75fe986e-063f-4101-8dee-3b49f2451d6a" (UID: "75fe986e-063f-4101-8dee-3b49f2451d6a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:43:46.079133 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.079105 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75fe986e-063f-4101-8dee-3b49f2451d6a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75fe986e-063f-4101-8dee-3b49f2451d6a" (UID: "75fe986e-063f-4101-8dee-3b49f2451d6a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:43:46.177664 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.177580 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75fe986e-063f-4101-8dee-3b49f2451d6a-openshift-service-ca-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:43:46.177664 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.177611 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75fe986e-063f-4101-8dee-3b49f2451d6a-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:43:46.378309 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.378270 2585 generic.go:356] "Generic (PLEG): container finished" podID="75fe986e-063f-4101-8dee-3b49f2451d6a" containerID="dc2a4812e0169836198c9b0678a7c3d09f12fbce1d79263edc974930bddc5eeb" exitCode=0 Jul 01 19:43:46.378477 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.378314 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" Jul 01 19:43:46.378477 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.378362 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" event={"ID":"75fe986e-063f-4101-8dee-3b49f2451d6a","Type":"ContainerDied","Data":"dc2a4812e0169836198c9b0678a7c3d09f12fbce1d79263edc974930bddc5eeb"} Jul 01 19:43:46.378477 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.378422 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs" event={"ID":"75fe986e-063f-4101-8dee-3b49f2451d6a","Type":"ContainerDied","Data":"4d1850c0bd71802874d89e3c96263d0dfe3fee41b09868602bd973579df78d42"} Jul 01 19:43:46.378477 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.378447 2585 scope.go:117] "RemoveContainer" containerID="dc2a4812e0169836198c9b0678a7c3d09f12fbce1d79263edc974930bddc5eeb" Jul 01 19:43:46.386936 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.386918 2585 scope.go:117] "RemoveContainer" containerID="dc2a4812e0169836198c9b0678a7c3d09f12fbce1d79263edc974930bddc5eeb" Jul 01 19:43:46.387188 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:43:46.387167 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc2a4812e0169836198c9b0678a7c3d09f12fbce1d79263edc974930bddc5eeb\": container with ID starting with dc2a4812e0169836198c9b0678a7c3d09f12fbce1d79263edc974930bddc5eeb not found: ID does not exist" containerID="dc2a4812e0169836198c9b0678a7c3d09f12fbce1d79263edc974930bddc5eeb" Jul 01 19:43:46.387250 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.387197 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2a4812e0169836198c9b0678a7c3d09f12fbce1d79263edc974930bddc5eeb"} err="failed to get container status \"dc2a4812e0169836198c9b0678a7c3d09f12fbce1d79263edc974930bddc5eeb\": rpc error: code = NotFound desc = could not find container \"dc2a4812e0169836198c9b0678a7c3d09f12fbce1d79263edc974930bddc5eeb\": container with ID starting with dc2a4812e0169836198c9b0678a7c3d09f12fbce1d79263edc974930bddc5eeb not found: ID does not exist" Jul 01 19:43:46.401912 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.401890 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs"] Jul 01 19:43:46.405330 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.405311 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-01ba2-68958d669b-8zsrs"] Jul 01 19:43:46.967031 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:46.966995 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75fe986e-063f-4101-8dee-3b49f2451d6a" path="/var/lib/kubelet/pods/75fe986e-063f-4101-8dee-3b49f2451d6a/volumes" Jul 01 19:43:48.255094 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:48.255042 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" podUID="1ba94df5-3413-40d5-834a-f447887dee1e" containerName="ensemble-graph-ac147" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:43:53.255048 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:53.255005 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" podUID="1ba94df5-3413-40d5-834a-f447887dee1e" containerName="ensemble-graph-ac147" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:43:56.423442 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:56.423408 2585 generic.go:356] "Generic (PLEG): container finished" podID="1ba94df5-3413-40d5-834a-f447887dee1e" containerID="b51f668157e427a201c00e183d8633f8c46643ff2d47293989dd1f5ea525a760" exitCode=0 Jul 01 19:43:56.423832 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:56.423485 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" event={"ID":"1ba94df5-3413-40d5-834a-f447887dee1e","Type":"ContainerDied","Data":"b51f668157e427a201c00e183d8633f8c46643ff2d47293989dd1f5ea525a760"} Jul 01 19:43:56.979928 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:56.979906 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" Jul 01 19:43:57.064910 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:57.064878 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba94df5-3413-40d5-834a-f447887dee1e-openshift-service-ca-bundle\") pod \"1ba94df5-3413-40d5-834a-f447887dee1e\" (UID: \"1ba94df5-3413-40d5-834a-f447887dee1e\") " Jul 01 19:43:57.065118 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:57.064922 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ba94df5-3413-40d5-834a-f447887dee1e-proxy-tls\") pod \"1ba94df5-3413-40d5-834a-f447887dee1e\" (UID: \"1ba94df5-3413-40d5-834a-f447887dee1e\") " Jul 01 19:43:57.065309 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:57.065284 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba94df5-3413-40d5-834a-f447887dee1e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1ba94df5-3413-40d5-834a-f447887dee1e" (UID: "1ba94df5-3413-40d5-834a-f447887dee1e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:43:57.067035 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:57.067013 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ba94df5-3413-40d5-834a-f447887dee1e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ba94df5-3413-40d5-834a-f447887dee1e" (UID: "1ba94df5-3413-40d5-834a-f447887dee1e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:43:57.166478 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:57.166387 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba94df5-3413-40d5-834a-f447887dee1e-openshift-service-ca-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:43:57.166478 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:57.166416 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ba94df5-3413-40d5-834a-f447887dee1e-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:43:57.428773 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:57.428697 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" Jul 01 19:43:57.428773 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:57.428735 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq" event={"ID":"1ba94df5-3413-40d5-834a-f447887dee1e","Type":"ContainerDied","Data":"ef111cb2d74902d2bfe2e018b08607b4242360429d6edc93233a2db31234a621"} Jul 01 19:43:57.429240 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:57.428794 2585 scope.go:117] "RemoveContainer" containerID="b51f668157e427a201c00e183d8633f8c46643ff2d47293989dd1f5ea525a760" Jul 01 19:43:57.450705 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:57.450679 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq"] Jul 01 19:43:57.454716 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:57.454698 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ac147-5f4c47599d-4q9vq"] Jul 01 19:43:58.962423 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:43:58.962389 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ba94df5-3413-40d5-834a-f447887dee1e" path="/var/lib/kubelet/pods/1ba94df5-3413-40d5-834a-f447887dee1e/volumes" Jul 01 19:44:16.031486 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.031447 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj"] Jul 01 19:44:16.032093 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.031970 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75fe986e-063f-4101-8dee-3b49f2451d6a" containerName="sequence-graph-01ba2" Jul 01 19:44:16.032093 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.031989 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="75fe986e-063f-4101-8dee-3b49f2451d6a" containerName="sequence-graph-01ba2" Jul 01 19:44:16.032093 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.032002 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ba94df5-3413-40d5-834a-f447887dee1e" containerName="ensemble-graph-ac147" Jul 01 19:44:16.032093 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.032011 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ba94df5-3413-40d5-834a-f447887dee1e" containerName="ensemble-graph-ac147" Jul 01 19:44:16.032323 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.032110 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ba94df5-3413-40d5-834a-f447887dee1e" containerName="ensemble-graph-ac147" Jul 01 19:44:16.032323 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.032123 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="75fe986e-063f-4101-8dee-3b49f2451d6a" containerName="sequence-graph-01ba2" Jul 01 19:44:16.036807 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.036787 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" Jul 01 19:44:16.037448 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.037420 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:44:16.039121 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.039102 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-46ddd-serving-cert\"" Jul 01 19:44:16.039121 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.039115 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-46ddd-kube-rbac-proxy-sar-config\"" Jul 01 19:44:16.039268 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.039135 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 01 19:44:16.039268 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.039158 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw4nc\"" Jul 01 19:44:16.043109 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.043091 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj"] Jul 01 19:44:16.128946 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.128910 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f27703-d667-45ea-8361-c30224333bff-openshift-service-ca-bundle\") pod \"sequence-graph-46ddd-6684c676db-rs8pj\" (UID: \"e0f27703-d667-45ea-8361-c30224333bff\") " pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" Jul 01 19:44:16.128946 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.128945 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f27703-d667-45ea-8361-c30224333bff-proxy-tls\") pod \"sequence-graph-46ddd-6684c676db-rs8pj\" (UID: \"e0f27703-d667-45ea-8361-c30224333bff\") " pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" Jul 01 19:44:16.156422 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.156389 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:44:16.229554 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.229521 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f27703-d667-45ea-8361-c30224333bff-openshift-service-ca-bundle\") pod \"sequence-graph-46ddd-6684c676db-rs8pj\" (UID: \"e0f27703-d667-45ea-8361-c30224333bff\") " pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" Jul 01 19:44:16.229554 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.229558 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f27703-d667-45ea-8361-c30224333bff-proxy-tls\") pod \"sequence-graph-46ddd-6684c676db-rs8pj\" (UID: \"e0f27703-d667-45ea-8361-c30224333bff\") " pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" Jul 01 19:44:16.230173 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.230153 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f27703-d667-45ea-8361-c30224333bff-openshift-service-ca-bundle\") pod \"sequence-graph-46ddd-6684c676db-rs8pj\" (UID: \"e0f27703-d667-45ea-8361-c30224333bff\") " pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" Jul 01 19:44:16.231945 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.231921 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f27703-d667-45ea-8361-c30224333bff-proxy-tls\") pod \"sequence-graph-46ddd-6684c676db-rs8pj\" (UID: \"e0f27703-d667-45ea-8361-c30224333bff\") " pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" Jul 01 19:44:16.347770 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.347687 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" Jul 01 19:44:16.474873 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.474840 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj"] Jul 01 19:44:16.477394 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:44:16.477356 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f27703_d667_45ea_8361_c30224333bff.slice/crio-2556443d4f840a3f591904e7fdf4d6d8fd50bb23ae0d29401f5387ced4d408e9 WatchSource:0}: Error finding container 2556443d4f840a3f591904e7fdf4d6d8fd50bb23ae0d29401f5387ced4d408e9: Status 404 returned error can't find the container with id 2556443d4f840a3f591904e7fdf4d6d8fd50bb23ae0d29401f5387ced4d408e9 Jul 01 19:44:16.519117 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:16.519092 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" event={"ID":"e0f27703-d667-45ea-8361-c30224333bff","Type":"ContainerStarted","Data":"2556443d4f840a3f591904e7fdf4d6d8fd50bb23ae0d29401f5387ced4d408e9"} Jul 01 19:44:17.525730 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:17.525693 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" event={"ID":"e0f27703-d667-45ea-8361-c30224333bff","Type":"ContainerStarted","Data":"0a3e360e2ef2fafeed73675e7fc0669fab6cc3009c527804c5b0e596fb7dad05"} Jul 01 19:44:17.526195 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:17.525821 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" Jul 01 19:44:17.542989 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:17.542942 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" podStartSLOduration=1.542929049 podStartE2EDuration="1.542929049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:44:17.542331276 +0000 UTC m=+1265.088707911" watchObservedRunningTime="2026-07-01 19:44:17.542929049 +0000 UTC m=+1265.089305682" Jul 01 19:44:23.536422 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:23.536393 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" Jul 01 19:44:26.102554 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.102497 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj"] Jul 01 19:44:26.103005 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.102773 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" podUID="e0f27703-d667-45ea-8361-c30224333bff" containerName="sequence-graph-46ddd" containerID="cri-o://0a3e360e2ef2fafeed73675e7fc0669fab6cc3009c527804c5b0e596fb7dad05" gracePeriod=30 Jul 01 19:44:26.124117 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.124083 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:44:26.210633 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.210576 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:44:26.307525 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.307489 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:44:26.406405 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.406315 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:44:26.527196 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.527169 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5"] Jul 01 19:44:26.530825 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.530799 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" Jul 01 19:44:26.533004 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.532982 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-c6a40-kube-rbac-proxy-sar-config\"" Jul 01 19:44:26.533154 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.533050 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-c6a40-serving-cert\"" Jul 01 19:44:26.538619 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.538585 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5"] Jul 01 19:44:26.541407 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.541380 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:44:26.644972 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.644869 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:44:26.713695 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.713617 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d77279e9-f056-42fc-bf5b-d4a3a951ee0c-openshift-service-ca-bundle\") pod \"ensemble-graph-c6a40-c7cbdccb5-rgtn5\" (UID: \"d77279e9-f056-42fc-bf5b-d4a3a951ee0c\") " pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" Jul 01 19:44:26.713695 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.713648 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77279e9-f056-42fc-bf5b-d4a3a951ee0c-proxy-tls\") pod \"ensemble-graph-c6a40-c7cbdccb5-rgtn5\" (UID: \"d77279e9-f056-42fc-bf5b-d4a3a951ee0c\") " pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" Jul 01 19:44:26.745081 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.745029 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:44:26.814355 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.814322 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d77279e9-f056-42fc-bf5b-d4a3a951ee0c-openshift-service-ca-bundle\") pod \"ensemble-graph-c6a40-c7cbdccb5-rgtn5\" (UID: \"d77279e9-f056-42fc-bf5b-d4a3a951ee0c\") " pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" Jul 01 19:44:26.814355 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.814357 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77279e9-f056-42fc-bf5b-d4a3a951ee0c-proxy-tls\") pod \"ensemble-graph-c6a40-c7cbdccb5-rgtn5\" (UID: \"d77279e9-f056-42fc-bf5b-d4a3a951ee0c\") " pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" Jul 01 19:44:26.814920 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.814901 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d77279e9-f056-42fc-bf5b-d4a3a951ee0c-openshift-service-ca-bundle\") pod \"ensemble-graph-c6a40-c7cbdccb5-rgtn5\" (UID: \"d77279e9-f056-42fc-bf5b-d4a3a951ee0c\") " pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" Jul 01 19:44:26.816751 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.816730 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77279e9-f056-42fc-bf5b-d4a3a951ee0c-proxy-tls\") pod \"ensemble-graph-c6a40-c7cbdccb5-rgtn5\" (UID: \"d77279e9-f056-42fc-bf5b-d4a3a951ee0c\") " pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" Jul 01 19:44:26.841867 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.841840 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" Jul 01 19:44:26.972156 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:26.972124 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5"] Jul 01 19:44:26.974229 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:44:26.974194 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77279e9_f056_42fc_bf5b_d4a3a951ee0c.slice/crio-5e26efed36eb0257ac76df5c40d6dcf012af62081fc46c5f1b15cb3acfa6ded1 WatchSource:0}: Error finding container 5e26efed36eb0257ac76df5c40d6dcf012af62081fc46c5f1b15cb3acfa6ded1: Status 404 returned error can't find the container with id 5e26efed36eb0257ac76df5c40d6dcf012af62081fc46c5f1b15cb3acfa6ded1 Jul 01 19:44:27.572676 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:27.572640 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" event={"ID":"d77279e9-f056-42fc-bf5b-d4a3a951ee0c","Type":"ContainerStarted","Data":"1fc33e8c6b9982b855b11875ae78a51db7bbd57f097c875feb5ff7300a2407ce"} Jul 01 19:44:27.573110 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:27.572697 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" event={"ID":"d77279e9-f056-42fc-bf5b-d4a3a951ee0c","Type":"ContainerStarted","Data":"5e26efed36eb0257ac76df5c40d6dcf012af62081fc46c5f1b15cb3acfa6ded1"} Jul 01 19:44:27.573110 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:27.572732 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" Jul 01 19:44:27.590837 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:27.590791 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" podStartSLOduration=1.590777441 podStartE2EDuration="1.590777441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:44:27.590382231 +0000 UTC m=+1275.136758864" watchObservedRunningTime="2026-07-01 19:44:27.590777441 +0000 UTC m=+1275.137154123" Jul 01 19:44:28.535103 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:28.535035 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" podUID="e0f27703-d667-45ea-8361-c30224333bff" containerName="sequence-graph-46ddd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:44:33.534746 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:33.534705 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" podUID="e0f27703-d667-45ea-8361-c30224333bff" containerName="sequence-graph-46ddd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:44:33.585433 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:33.585404 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" Jul 01 19:44:38.534976 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:38.534937 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" podUID="e0f27703-d667-45ea-8361-c30224333bff" containerName="sequence-graph-46ddd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:44:38.535469 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:38.535040 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" Jul 01 19:44:43.535080 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:43.535026 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" podUID="e0f27703-d667-45ea-8361-c30224333bff" containerName="sequence-graph-46ddd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:44:48.534833 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:48.534788 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" podUID="e0f27703-d667-45ea-8361-c30224333bff" containerName="sequence-graph-46ddd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:44:53.534286 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:53.534250 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" podUID="e0f27703-d667-45ea-8361-c30224333bff" containerName="sequence-graph-46ddd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:44:56.253648 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.253624 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" Jul 01 19:44:56.354209 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.354173 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f27703-d667-45ea-8361-c30224333bff-proxy-tls\") pod \"e0f27703-d667-45ea-8361-c30224333bff\" (UID: \"e0f27703-d667-45ea-8361-c30224333bff\") " Jul 01 19:44:56.354209 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.354214 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f27703-d667-45ea-8361-c30224333bff-openshift-service-ca-bundle\") pod \"e0f27703-d667-45ea-8361-c30224333bff\" (UID: \"e0f27703-d667-45ea-8361-c30224333bff\") " Jul 01 19:44:56.354563 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.354539 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f27703-d667-45ea-8361-c30224333bff-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e0f27703-d667-45ea-8361-c30224333bff" (UID: "e0f27703-d667-45ea-8361-c30224333bff"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:44:56.356212 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.356192 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f27703-d667-45ea-8361-c30224333bff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0f27703-d667-45ea-8361-c30224333bff" (UID: "e0f27703-d667-45ea-8361-c30224333bff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:44:56.455466 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.455428 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f27703-d667-45ea-8361-c30224333bff-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:44:56.455466 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.455457 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f27703-d667-45ea-8361-c30224333bff-openshift-service-ca-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:44:56.707753 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.707662 2585 generic.go:356] "Generic (PLEG): container finished" podID="e0f27703-d667-45ea-8361-c30224333bff" containerID="0a3e360e2ef2fafeed73675e7fc0669fab6cc3009c527804c5b0e596fb7dad05" exitCode=0 Jul 01 19:44:56.707753 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.707700 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" Jul 01 19:44:56.707753 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.707746 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" event={"ID":"e0f27703-d667-45ea-8361-c30224333bff","Type":"ContainerDied","Data":"0a3e360e2ef2fafeed73675e7fc0669fab6cc3009c527804c5b0e596fb7dad05"} Jul 01 19:44:56.708015 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.707828 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj" event={"ID":"e0f27703-d667-45ea-8361-c30224333bff","Type":"ContainerDied","Data":"2556443d4f840a3f591904e7fdf4d6d8fd50bb23ae0d29401f5387ced4d408e9"} Jul 01 19:44:56.708015 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.707850 2585 scope.go:117] "RemoveContainer" containerID="0a3e360e2ef2fafeed73675e7fc0669fab6cc3009c527804c5b0e596fb7dad05" Jul 01 19:44:56.716430 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.716408 2585 scope.go:117] "RemoveContainer" containerID="0a3e360e2ef2fafeed73675e7fc0669fab6cc3009c527804c5b0e596fb7dad05" Jul 01 19:44:56.716677 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:44:56.716659 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3e360e2ef2fafeed73675e7fc0669fab6cc3009c527804c5b0e596fb7dad05\": container with ID starting with 0a3e360e2ef2fafeed73675e7fc0669fab6cc3009c527804c5b0e596fb7dad05 not found: ID does not exist" containerID="0a3e360e2ef2fafeed73675e7fc0669fab6cc3009c527804c5b0e596fb7dad05" Jul 01 19:44:56.716722 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.716685 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3e360e2ef2fafeed73675e7fc0669fab6cc3009c527804c5b0e596fb7dad05"} err="failed to get container status \"0a3e360e2ef2fafeed73675e7fc0669fab6cc3009c527804c5b0e596fb7dad05\": rpc error: code = NotFound desc = could not find container \"0a3e360e2ef2fafeed73675e7fc0669fab6cc3009c527804c5b0e596fb7dad05\": container with ID starting with 0a3e360e2ef2fafeed73675e7fc0669fab6cc3009c527804c5b0e596fb7dad05 not found: ID does not exist" Jul 01 19:44:56.729707 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.729683 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj"] Jul 01 19:44:56.731185 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.731167 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-46ddd-6684c676db-rs8pj"] Jul 01 19:44:56.961592 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:44:56.961502 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0f27703-d667-45ea-8361-c30224333bff" path="/var/lib/kubelet/pods/e0f27703-d667-45ea-8361-c30224333bff/volumes" Jul 01 19:45:06.007028 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:06.006990 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:45:26.308900 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.308867 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd"] Jul 01 19:45:26.309393 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.309253 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f27703-d667-45ea-8361-c30224333bff" containerName="sequence-graph-46ddd" Jul 01 19:45:26.309393 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.309266 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f27703-d667-45ea-8361-c30224333bff" containerName="sequence-graph-46ddd" Jul 01 19:45:26.309393 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.309342 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0f27703-d667-45ea-8361-c30224333bff" containerName="sequence-graph-46ddd" Jul 01 19:45:26.312301 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.312286 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" Jul 01 19:45:26.314158 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.314119 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:45:26.314739 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.314720 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-88843-kube-rbac-proxy-sar-config\"" Jul 01 19:45:26.314870 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.314723 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-88843-serving-cert\"" Jul 01 19:45:26.323549 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.323526 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd"] Jul 01 19:45:26.390465 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.390431 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:45:26.405660 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.405632 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e81ab7d-29e1-46b0-997c-542f6571299c-proxy-tls\") pod \"sequence-graph-88843-5799f5fbfd-cbgsd\" (UID: \"6e81ab7d-29e1-46b0-997c-542f6571299c\") " pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" Jul 01 19:45:26.405804 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.405669 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e81ab7d-29e1-46b0-997c-542f6571299c-openshift-service-ca-bundle\") pod \"sequence-graph-88843-5799f5fbfd-cbgsd\" (UID: \"6e81ab7d-29e1-46b0-997c-542f6571299c\") " pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" Jul 01 19:45:26.506480 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.506447 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e81ab7d-29e1-46b0-997c-542f6571299c-proxy-tls\") pod \"sequence-graph-88843-5799f5fbfd-cbgsd\" (UID: \"6e81ab7d-29e1-46b0-997c-542f6571299c\") " pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" Jul 01 19:45:26.506640 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.506488 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e81ab7d-29e1-46b0-997c-542f6571299c-openshift-service-ca-bundle\") pod \"sequence-graph-88843-5799f5fbfd-cbgsd\" (UID: \"6e81ab7d-29e1-46b0-997c-542f6571299c\") " pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" Jul 01 19:45:26.507078 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.507050 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e81ab7d-29e1-46b0-997c-542f6571299c-openshift-service-ca-bundle\") pod \"sequence-graph-88843-5799f5fbfd-cbgsd\" (UID: \"6e81ab7d-29e1-46b0-997c-542f6571299c\") " pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" Jul 01 19:45:26.508758 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.508744 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e81ab7d-29e1-46b0-997c-542f6571299c-proxy-tls\") pod \"sequence-graph-88843-5799f5fbfd-cbgsd\" (UID: \"6e81ab7d-29e1-46b0-997c-542f6571299c\") " pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" Jul 01 19:45:26.622804 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.622703 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" Jul 01 19:45:26.754200 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.754177 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd"] Jul 01 19:45:26.756231 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:45:26.756202 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e81ab7d_29e1_46b0_997c_542f6571299c.slice/crio-c9a2498b73586e03db222b5ef910965d3d1009b5333fdf0aee64764c111adea4 WatchSource:0}: Error finding container c9a2498b73586e03db222b5ef910965d3d1009b5333fdf0aee64764c111adea4: Status 404 returned error can't find the container with id c9a2498b73586e03db222b5ef910965d3d1009b5333fdf0aee64764c111adea4 Jul 01 19:45:26.839293 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.839247 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" event={"ID":"6e81ab7d-29e1-46b0-997c-542f6571299c","Type":"ContainerStarted","Data":"da3a97a326c1f135f6a162ab9007a5354d2e21ad8a937d6a4b1b2ad9237211f5"} Jul 01 19:45:26.839460 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.839309 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" event={"ID":"6e81ab7d-29e1-46b0-997c-542f6571299c","Type":"ContainerStarted","Data":"c9a2498b73586e03db222b5ef910965d3d1009b5333fdf0aee64764c111adea4"} Jul 01 19:45:26.839460 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.839396 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" Jul 01 19:45:26.855276 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:26.855227 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" podStartSLOduration=0.855212059 podStartE2EDuration="855.212059ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:45:26.854096431 +0000 UTC m=+1334.400473057" watchObservedRunningTime="2026-07-01 19:45:26.855212059 +0000 UTC m=+1334.401588693" Jul 01 19:45:32.849006 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:45:32.848979 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" Jul 01 19:46:21.693413 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:46:21.693370 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:46:33.370750 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:46:33.370707 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:48:12.975741 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:48:12.975714 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:48:12.978883 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:48:12.978863 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:51:35.171342 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:51:35.171266 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:52:41.048994 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:52:41.048957 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5"] Jul 01 19:52:41.049610 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:52:41.049240 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" podUID="d77279e9-f056-42fc-bf5b-d4a3a951ee0c" containerName="ensemble-graph-c6a40" containerID="cri-o://1fc33e8c6b9982b855b11875ae78a51db7bbd57f097c875feb5ff7300a2407ce" gracePeriod=30 Jul 01 19:52:41.070487 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:52:41.070453 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:52:41.249966 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:52:41.249926 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:52:41.345224 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:52:41.345140 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:52:41.436105 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:52:41.436053 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:52:41.501831 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:52:41.501794 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:52:41.603651 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:52:41.603549 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:52:43.583159 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:52:43.583124 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" podUID="d77279e9-f056-42fc-bf5b-d4a3a951ee0c" containerName="ensemble-graph-c6a40" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:52:48.583274 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:52:48.583233 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" podUID="d77279e9-f056-42fc-bf5b-d4a3a951ee0c" containerName="ensemble-graph-c6a40" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:52:53.583442 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:52:53.583401 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" podUID="d77279e9-f056-42fc-bf5b-d4a3a951ee0c" containerName="ensemble-graph-c6a40" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:52:53.583911 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:52:53.583500 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" Jul 01 19:52:58.583705 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:52:58.583663 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" podUID="d77279e9-f056-42fc-bf5b-d4a3a951ee0c" containerName="ensemble-graph-c6a40" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:53:03.583134 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:03.583093 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" podUID="d77279e9-f056-42fc-bf5b-d4a3a951ee0c" containerName="ensemble-graph-c6a40" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:53:08.583050 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:08.582970 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" podUID="d77279e9-f056-42fc-bf5b-d4a3a951ee0c" containerName="ensemble-graph-c6a40" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:53:11.202346 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.202322 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" Jul 01 19:53:11.251940 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.251903 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77279e9-f056-42fc-bf5b-d4a3a951ee0c-proxy-tls\") pod \"d77279e9-f056-42fc-bf5b-d4a3a951ee0c\" (UID: \"d77279e9-f056-42fc-bf5b-d4a3a951ee0c\") " Jul 01 19:53:11.252140 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.252023 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d77279e9-f056-42fc-bf5b-d4a3a951ee0c-openshift-service-ca-bundle\") pod \"d77279e9-f056-42fc-bf5b-d4a3a951ee0c\" (UID: \"d77279e9-f056-42fc-bf5b-d4a3a951ee0c\") " Jul 01 19:53:11.252379 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.252354 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77279e9-f056-42fc-bf5b-d4a3a951ee0c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d77279e9-f056-42fc-bf5b-d4a3a951ee0c" (UID: "d77279e9-f056-42fc-bf5b-d4a3a951ee0c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:53:11.254021 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.254000 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d77279e9-f056-42fc-bf5b-d4a3a951ee0c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d77279e9-f056-42fc-bf5b-d4a3a951ee0c" (UID: "d77279e9-f056-42fc-bf5b-d4a3a951ee0c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:53:11.353338 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.353250 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d77279e9-f056-42fc-bf5b-d4a3a951ee0c-openshift-service-ca-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:53:11.353338 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.353281 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77279e9-f056-42fc-bf5b-d4a3a951ee0c-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:53:11.887591 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.887554 2585 generic.go:356] "Generic (PLEG): container finished" podID="d77279e9-f056-42fc-bf5b-d4a3a951ee0c" containerID="1fc33e8c6b9982b855b11875ae78a51db7bbd57f097c875feb5ff7300a2407ce" exitCode=0 Jul 01 19:53:11.887770 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.887614 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" Jul 01 19:53:11.887770 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.887641 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" event={"ID":"d77279e9-f056-42fc-bf5b-d4a3a951ee0c","Type":"ContainerDied","Data":"1fc33e8c6b9982b855b11875ae78a51db7bbd57f097c875feb5ff7300a2407ce"} Jul 01 19:53:11.887770 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.887723 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5" event={"ID":"d77279e9-f056-42fc-bf5b-d4a3a951ee0c","Type":"ContainerDied","Data":"5e26efed36eb0257ac76df5c40d6dcf012af62081fc46c5f1b15cb3acfa6ded1"} Jul 01 19:53:11.887770 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.887750 2585 scope.go:117] "RemoveContainer" containerID="1fc33e8c6b9982b855b11875ae78a51db7bbd57f097c875feb5ff7300a2407ce" Jul 01 19:53:11.896359 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.896342 2585 scope.go:117] "RemoveContainer" containerID="1fc33e8c6b9982b855b11875ae78a51db7bbd57f097c875feb5ff7300a2407ce" Jul 01 19:53:11.896638 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:53:11.896615 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc33e8c6b9982b855b11875ae78a51db7bbd57f097c875feb5ff7300a2407ce\": container with ID starting with 1fc33e8c6b9982b855b11875ae78a51db7bbd57f097c875feb5ff7300a2407ce not found: ID does not exist" containerID="1fc33e8c6b9982b855b11875ae78a51db7bbd57f097c875feb5ff7300a2407ce" Jul 01 19:53:11.896698 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.896648 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc33e8c6b9982b855b11875ae78a51db7bbd57f097c875feb5ff7300a2407ce"} err="failed to get container status \"1fc33e8c6b9982b855b11875ae78a51db7bbd57f097c875feb5ff7300a2407ce\": rpc error: code = NotFound desc = could not find container \"1fc33e8c6b9982b855b11875ae78a51db7bbd57f097c875feb5ff7300a2407ce\": container with ID starting with 1fc33e8c6b9982b855b11875ae78a51db7bbd57f097c875feb5ff7300a2407ce not found: ID does not exist" Jul 01 19:53:11.908834 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.908811 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5"] Jul 01 19:53:11.913174 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:11.913151 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c6a40-c7cbdccb5-rgtn5"] Jul 01 19:53:12.962513 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:12.962475 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d77279e9-f056-42fc-bf5b-d4a3a951ee0c" path="/var/lib/kubelet/pods/d77279e9-f056-42fc-bf5b-d4a3a951ee0c/volumes" Jul 01 19:53:13.000953 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:13.000928 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:53:13.005102 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:13.005077 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:53:41.039472 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.039438 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd"] Jul 01 19:53:41.039854 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.039674 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" podUID="6e81ab7d-29e1-46b0-997c-542f6571299c" containerName="sequence-graph-88843" containerID="cri-o://da3a97a326c1f135f6a162ab9007a5354d2e21ad8a937d6a4b1b2ad9237211f5" gracePeriod=30 Jul 01 19:53:41.055008 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.054962 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:53:41.173400 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.173364 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:53:41.274741 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.274709 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:53:41.320181 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.320050 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8"] Jul 01 19:53:41.320428 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.320415 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d77279e9-f056-42fc-bf5b-d4a3a951ee0c" containerName="ensemble-graph-c6a40" Jul 01 19:53:41.320482 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.320429 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d77279e9-f056-42fc-bf5b-d4a3a951ee0c" containerName="ensemble-graph-c6a40" Jul 01 19:53:41.320524 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.320489 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d77279e9-f056-42fc-bf5b-d4a3a951ee0c" containerName="ensemble-graph-c6a40" Jul 01 19:53:41.323423 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.323408 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" Jul 01 19:53:41.327023 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.326990 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-74731-kube-rbac-proxy-sar-config\"" Jul 01 19:53:41.327574 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.327553 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-74731-serving-cert\"" Jul 01 19:53:41.335819 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.335796 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8"] Jul 01 19:53:41.372833 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.372790 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:53:41.513319 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.513281 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe83d95f-4730-40b4-be32-ac9052550795-openshift-service-ca-bundle\") pod \"splitter-graph-74731-594747d957-djfc8\" (UID: \"fe83d95f-4730-40b4-be32-ac9052550795\") " pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" Jul 01 19:53:41.513485 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.513342 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe83d95f-4730-40b4-be32-ac9052550795-proxy-tls\") pod \"splitter-graph-74731-594747d957-djfc8\" (UID: \"fe83d95f-4730-40b4-be32-ac9052550795\") " pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" Jul 01 19:53:41.525253 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.525219 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:53:41.613989 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.613897 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe83d95f-4730-40b4-be32-ac9052550795-openshift-service-ca-bundle\") pod \"splitter-graph-74731-594747d957-djfc8\" (UID: \"fe83d95f-4730-40b4-be32-ac9052550795\") " pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" Jul 01 19:53:41.613989 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.613958 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe83d95f-4730-40b4-be32-ac9052550795-proxy-tls\") pod \"splitter-graph-74731-594747d957-djfc8\" (UID: \"fe83d95f-4730-40b4-be32-ac9052550795\") " pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" Jul 01 19:53:41.614234 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:53:41.614098 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-74731-serving-cert: secret "splitter-graph-74731-serving-cert" not found Jul 01 19:53:41.614234 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:53:41.614165 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe83d95f-4730-40b4-be32-ac9052550795-proxy-tls podName:fe83d95f-4730-40b4-be32-ac9052550795 nodeName:}" failed. No retries permitted until 2026-07-01 19:53:42.114149266 +0000 UTC m=+1829.660525877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fe83d95f-4730-40b4-be32-ac9052550795-proxy-tls") pod "splitter-graph-74731-594747d957-djfc8" (UID: "fe83d95f-4730-40b4-be32-ac9052550795") : secret "splitter-graph-74731-serving-cert" not found Jul 01 19:53:41.614644 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.614624 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe83d95f-4730-40b4-be32-ac9052550795-openshift-service-ca-bundle\") pod \"splitter-graph-74731-594747d957-djfc8\" (UID: \"fe83d95f-4730-40b4-be32-ac9052550795\") " pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" Jul 01 19:53:41.714948 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:41.714910 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:53:42.117874 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:42.117832 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe83d95f-4730-40b4-be32-ac9052550795-proxy-tls\") pod \"splitter-graph-74731-594747d957-djfc8\" (UID: \"fe83d95f-4730-40b4-be32-ac9052550795\") " pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" Jul 01 19:53:42.120277 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:42.120255 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe83d95f-4730-40b4-be32-ac9052550795-proxy-tls\") pod \"splitter-graph-74731-594747d957-djfc8\" (UID: \"fe83d95f-4730-40b4-be32-ac9052550795\") " pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" Jul 01 19:53:42.234714 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:42.234649 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" Jul 01 19:53:42.363355 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:42.363330 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8"] Jul 01 19:53:42.365861 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:53:42.365829 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe83d95f_4730_40b4_be32_ac9052550795.slice/crio-3b6088317a26a7cbd45929beb9d5e8ebfd873eb840ed4d5349388e4b189e4f6d WatchSource:0}: Error finding container 3b6088317a26a7cbd45929beb9d5e8ebfd873eb840ed4d5349388e4b189e4f6d: Status 404 returned error can't find the container with id 3b6088317a26a7cbd45929beb9d5e8ebfd873eb840ed4d5349388e4b189e4f6d Jul 01 19:53:42.848245 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:42.848210 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" podUID="6e81ab7d-29e1-46b0-997c-542f6571299c" containerName="sequence-graph-88843" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:53:43.031381 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:43.031341 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" event={"ID":"fe83d95f-4730-40b4-be32-ac9052550795","Type":"ContainerStarted","Data":"f89afaec219ca447ffa4046f861b9ad343b956a016d694dd703e92e05dbfffc6"} Jul 01 19:53:43.031558 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:43.031395 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" event={"ID":"fe83d95f-4730-40b4-be32-ac9052550795","Type":"ContainerStarted","Data":"3b6088317a26a7cbd45929beb9d5e8ebfd873eb840ed4d5349388e4b189e4f6d"} Jul 01 19:53:43.031558 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:43.031441 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" Jul 01 19:53:43.059544 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:43.059489 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" podStartSLOduration=2.059472474 podStartE2EDuration="2.059472474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:53:43.057830009 +0000 UTC m=+1830.604206646" watchObservedRunningTime="2026-07-01 19:53:43.059472474 +0000 UTC m=+1830.605849109" Jul 01 19:53:47.847836 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:47.847787 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" podUID="6e81ab7d-29e1-46b0-997c-542f6571299c" containerName="sequence-graph-88843" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:53:49.042398 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:49.042372 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" Jul 01 19:53:51.379346 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:51.379269 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8"] Jul 01 19:53:51.379747 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:51.379452 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" podUID="fe83d95f-4730-40b4-be32-ac9052550795" containerName="splitter-graph-74731" containerID="cri-o://f89afaec219ca447ffa4046f861b9ad343b956a016d694dd703e92e05dbfffc6" gracePeriod=30 Jul 01 19:53:51.410557 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:51.410528 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:53:51.613491 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:51.613456 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:53:51.710866 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:51.710786 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:53:51.807985 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:51.807955 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:53:51.872286 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:51.872249 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:53:51.988473 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:51.988396 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:53:52.847682 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:52.847646 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" podUID="6e81ab7d-29e1-46b0-997c-542f6571299c" containerName="sequence-graph-88843" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:53:52.848086 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:52.847755 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" Jul 01 19:53:54.040557 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:54.040523 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" podUID="fe83d95f-4730-40b4-be32-ac9052550795" containerName="splitter-graph-74731" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:53:57.848013 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:57.847976 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" podUID="6e81ab7d-29e1-46b0-997c-542f6571299c" containerName="sequence-graph-88843" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:53:59.040207 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:53:59.040165 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" podUID="fe83d95f-4730-40b4-be32-ac9052550795" containerName="splitter-graph-74731" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:54:02.847803 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:02.847763 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" podUID="6e81ab7d-29e1-46b0-997c-542f6571299c" containerName="sequence-graph-88843" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:54:04.040121 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:04.040080 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" podUID="fe83d95f-4730-40b4-be32-ac9052550795" containerName="splitter-graph-74731" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:54:04.040495 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:04.040191 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" Jul 01 19:54:07.848084 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:07.848034 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" podUID="6e81ab7d-29e1-46b0-997c-542f6571299c" containerName="sequence-graph-88843" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:54:09.040331 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:09.040295 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" podUID="fe83d95f-4730-40b4-be32-ac9052550795" containerName="splitter-graph-74731" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:54:11.158984 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:11.158942 2585 generic.go:356] "Generic (PLEG): container finished" podID="6e81ab7d-29e1-46b0-997c-542f6571299c" containerID="da3a97a326c1f135f6a162ab9007a5354d2e21ad8a937d6a4b1b2ad9237211f5" exitCode=0 Jul 01 19:54:11.159364 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:11.159012 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" event={"ID":"6e81ab7d-29e1-46b0-997c-542f6571299c","Type":"ContainerDied","Data":"da3a97a326c1f135f6a162ab9007a5354d2e21ad8a937d6a4b1b2ad9237211f5"} Jul 01 19:54:11.201005 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:11.200960 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" Jul 01 19:54:11.261965 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:11.261928 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e81ab7d-29e1-46b0-997c-542f6571299c-openshift-service-ca-bundle\") pod \"6e81ab7d-29e1-46b0-997c-542f6571299c\" (UID: \"6e81ab7d-29e1-46b0-997c-542f6571299c\") " Jul 01 19:54:11.262156 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:11.261975 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e81ab7d-29e1-46b0-997c-542f6571299c-proxy-tls\") pod \"6e81ab7d-29e1-46b0-997c-542f6571299c\" (UID: \"6e81ab7d-29e1-46b0-997c-542f6571299c\") " Jul 01 19:54:11.262404 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:11.262376 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e81ab7d-29e1-46b0-997c-542f6571299c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6e81ab7d-29e1-46b0-997c-542f6571299c" (UID: "6e81ab7d-29e1-46b0-997c-542f6571299c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:54:11.264158 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:11.264130 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e81ab7d-29e1-46b0-997c-542f6571299c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e81ab7d-29e1-46b0-997c-542f6571299c" (UID: "6e81ab7d-29e1-46b0-997c-542f6571299c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:54:11.363095 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:11.362985 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e81ab7d-29e1-46b0-997c-542f6571299c-openshift-service-ca-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:54:11.363095 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:11.363017 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e81ab7d-29e1-46b0-997c-542f6571299c-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:54:12.164241 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:12.164209 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" Jul 01 19:54:12.164647 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:12.164247 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd" event={"ID":"6e81ab7d-29e1-46b0-997c-542f6571299c","Type":"ContainerDied","Data":"c9a2498b73586e03db222b5ef910965d3d1009b5333fdf0aee64764c111adea4"} Jul 01 19:54:12.164647 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:12.164308 2585 scope.go:117] "RemoveContainer" containerID="da3a97a326c1f135f6a162ab9007a5354d2e21ad8a937d6a4b1b2ad9237211f5" Jul 01 19:54:12.186327 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:12.186296 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd"] Jul 01 19:54:12.190502 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:12.190482 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-88843-5799f5fbfd-cbgsd"] Jul 01 19:54:12.964230 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:12.964199 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e81ab7d-29e1-46b0-997c-542f6571299c" path="/var/lib/kubelet/pods/6e81ab7d-29e1-46b0-997c-542f6571299c/volumes" Jul 01 19:54:14.040689 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:14.040649 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" podUID="fe83d95f-4730-40b4-be32-ac9052550795" containerName="splitter-graph-74731" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:54:19.040208 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:19.040172 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" podUID="fe83d95f-4730-40b4-be32-ac9052550795" containerName="splitter-graph-74731" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 19:54:21.532310 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:21.532284 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" Jul 01 19:54:21.644029 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:21.643941 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe83d95f-4730-40b4-be32-ac9052550795-openshift-service-ca-bundle\") pod \"fe83d95f-4730-40b4-be32-ac9052550795\" (UID: \"fe83d95f-4730-40b4-be32-ac9052550795\") " Jul 01 19:54:21.644201 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:21.644022 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe83d95f-4730-40b4-be32-ac9052550795-proxy-tls\") pod \"fe83d95f-4730-40b4-be32-ac9052550795\" (UID: \"fe83d95f-4730-40b4-be32-ac9052550795\") " Jul 01 19:54:21.644412 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:21.644390 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe83d95f-4730-40b4-be32-ac9052550795-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fe83d95f-4730-40b4-be32-ac9052550795" (UID: "fe83d95f-4730-40b4-be32-ac9052550795"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 19:54:21.646113 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:21.646089 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe83d95f-4730-40b4-be32-ac9052550795-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe83d95f-4730-40b4-be32-ac9052550795" (UID: "fe83d95f-4730-40b4-be32-ac9052550795"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 19:54:21.745171 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:21.745135 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe83d95f-4730-40b4-be32-ac9052550795-openshift-service-ca-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:54:21.745171 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:21.745165 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe83d95f-4730-40b4-be32-ac9052550795-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 19:54:22.208425 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:22.208393 2585 generic.go:356] "Generic (PLEG): container finished" podID="fe83d95f-4730-40b4-be32-ac9052550795" containerID="f89afaec219ca447ffa4046f861b9ad343b956a016d694dd703e92e05dbfffc6" exitCode=0 Jul 01 19:54:22.208612 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:22.208436 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" Jul 01 19:54:22.208612 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:22.208486 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" event={"ID":"fe83d95f-4730-40b4-be32-ac9052550795","Type":"ContainerDied","Data":"f89afaec219ca447ffa4046f861b9ad343b956a016d694dd703e92e05dbfffc6"} Jul 01 19:54:22.208612 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:22.208536 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8" event={"ID":"fe83d95f-4730-40b4-be32-ac9052550795","Type":"ContainerDied","Data":"3b6088317a26a7cbd45929beb9d5e8ebfd873eb840ed4d5349388e4b189e4f6d"} Jul 01 19:54:22.208612 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:22.208558 2585 scope.go:117] "RemoveContainer" containerID="f89afaec219ca447ffa4046f861b9ad343b956a016d694dd703e92e05dbfffc6" Jul 01 19:54:22.219726 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:22.219693 2585 scope.go:117] "RemoveContainer" containerID="f89afaec219ca447ffa4046f861b9ad343b956a016d694dd703e92e05dbfffc6" Jul 01 19:54:22.220133 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:54:22.220105 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89afaec219ca447ffa4046f861b9ad343b956a016d694dd703e92e05dbfffc6\": container with ID starting with f89afaec219ca447ffa4046f861b9ad343b956a016d694dd703e92e05dbfffc6 not found: ID does not exist" containerID="f89afaec219ca447ffa4046f861b9ad343b956a016d694dd703e92e05dbfffc6" Jul 01 19:54:22.220249 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:22.220142 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89afaec219ca447ffa4046f861b9ad343b956a016d694dd703e92e05dbfffc6"} err="failed to get container status \"f89afaec219ca447ffa4046f861b9ad343b956a016d694dd703e92e05dbfffc6\": rpc error: code = NotFound desc = could not find container \"f89afaec219ca447ffa4046f861b9ad343b956a016d694dd703e92e05dbfffc6\": container with ID starting with f89afaec219ca447ffa4046f861b9ad343b956a016d694dd703e92e05dbfffc6 not found: ID does not exist" Jul 01 19:54:22.230986 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:22.230961 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8"] Jul 01 19:54:22.235261 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:22.235237 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-74731-594747d957-djfc8"] Jul 01 19:54:22.961167 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:22.961134 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe83d95f-4730-40b4-be32-ac9052550795" path="/var/lib/kubelet/pods/fe83d95f-4730-40b4-be32-ac9052550795/volumes" Jul 01 19:54:41.250725 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.250692 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w"] Jul 01 19:54:41.251148 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.251017 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe83d95f-4730-40b4-be32-ac9052550795" containerName="splitter-graph-74731" Jul 01 19:54:41.251148 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.251029 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe83d95f-4730-40b4-be32-ac9052550795" containerName="splitter-graph-74731" Jul 01 19:54:41.251148 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.251044 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e81ab7d-29e1-46b0-997c-542f6571299c" containerName="sequence-graph-88843" Jul 01 19:54:41.251148 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.251049 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e81ab7d-29e1-46b0-997c-542f6571299c" containerName="sequence-graph-88843" Jul 01 19:54:41.251148 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.251113 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe83d95f-4730-40b4-be32-ac9052550795" containerName="splitter-graph-74731" Jul 01 19:54:41.251148 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.251123 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e81ab7d-29e1-46b0-997c-542f6571299c" containerName="sequence-graph-88843" Jul 01 19:54:41.255294 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.255273 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" Jul 01 19:54:41.259190 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.259160 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ee34f-serving-cert\"" Jul 01 19:54:41.259387 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.259371 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ee34f-kube-rbac-proxy-sar-config\"" Jul 01 19:54:41.259776 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.259757 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 01 19:54:41.259892 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.259792 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw4nc\"" Jul 01 19:54:41.279912 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.279882 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w"] Jul 01 19:54:41.303897 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.303856 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:54:41.405005 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.404968 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc929f5f-8d42-4360-9329-a4b81c206f7f-proxy-tls\") pod \"switch-graph-ee34f-5765b8d4c6-r9x5w\" (UID: \"fc929f5f-8d42-4360-9329-a4b81c206f7f\") " pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" Jul 01 19:54:41.405217 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.405041 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc929f5f-8d42-4360-9329-a4b81c206f7f-openshift-service-ca-bundle\") pod \"switch-graph-ee34f-5765b8d4c6-r9x5w\" (UID: \"fc929f5f-8d42-4360-9329-a4b81c206f7f\") " pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" Jul 01 19:54:41.460637 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.460582 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:54:41.506105 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.505970 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc929f5f-8d42-4360-9329-a4b81c206f7f-proxy-tls\") pod \"switch-graph-ee34f-5765b8d4c6-r9x5w\" (UID: \"fc929f5f-8d42-4360-9329-a4b81c206f7f\") " pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" Jul 01 19:54:41.506105 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.506036 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc929f5f-8d42-4360-9329-a4b81c206f7f-openshift-service-ca-bundle\") pod \"switch-graph-ee34f-5765b8d4c6-r9x5w\" (UID: \"fc929f5f-8d42-4360-9329-a4b81c206f7f\") " pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" Jul 01 19:54:41.506775 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.506746 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc929f5f-8d42-4360-9329-a4b81c206f7f-openshift-service-ca-bundle\") pod \"switch-graph-ee34f-5765b8d4c6-r9x5w\" (UID: \"fc929f5f-8d42-4360-9329-a4b81c206f7f\") " pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" Jul 01 19:54:41.508459 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.508440 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc929f5f-8d42-4360-9329-a4b81c206f7f-proxy-tls\") pod \"switch-graph-ee34f-5765b8d4c6-r9x5w\" (UID: \"fc929f5f-8d42-4360-9329-a4b81c206f7f\") " pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" Jul 01 19:54:41.565217 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.565181 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" Jul 01 19:54:41.694864 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:41.694836 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w"] Jul 01 19:54:41.697257 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:54:41.697228 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc929f5f_8d42_4360_9329_a4b81c206f7f.slice/crio-384b960facdf4c0c4b6a22d06ae2aeedd1ffae63313ea5fee9064538a27014c4 WatchSource:0}: Error finding container 384b960facdf4c0c4b6a22d06ae2aeedd1ffae63313ea5fee9064538a27014c4: Status 404 returned error can't find the container with id 384b960facdf4c0c4b6a22d06ae2aeedd1ffae63313ea5fee9064538a27014c4 Jul 01 19:54:42.297436 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:42.297401 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" event={"ID":"fc929f5f-8d42-4360-9329-a4b81c206f7f","Type":"ContainerStarted","Data":"41aa0cba55edbc509ed34cb1e37108ef9fd1178aaef4028fc90603c015e41490"} Jul 01 19:54:42.297895 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:42.297449 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" event={"ID":"fc929f5f-8d42-4360-9329-a4b81c206f7f","Type":"ContainerStarted","Data":"384b960facdf4c0c4b6a22d06ae2aeedd1ffae63313ea5fee9064538a27014c4"} Jul 01 19:54:42.297895 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:42.297534 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" Jul 01 19:54:42.316899 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:42.316846 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" podStartSLOduration=1.316827543 podStartE2EDuration="1.316827543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:54:42.315953293 +0000 UTC m=+1889.862329940" watchObservedRunningTime="2026-07-01 19:54:42.316827543 +0000 UTC m=+1889.863204176" Jul 01 19:54:48.308266 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:48.308238 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" Jul 01 19:54:51.584183 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:51.584150 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4"] Jul 01 19:54:51.587430 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:51.587410 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" Jul 01 19:54:51.589169 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:51.589144 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-openshift-service-ca-bundle\") pod \"splitter-graph-b4182-58cdcd74d5-w67h4\" (UID: \"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8\") " pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" Jul 01 19:54:51.589264 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:51.589203 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-proxy-tls\") pod \"splitter-graph-b4182-58cdcd74d5-w67h4\" (UID: \"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8\") " pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" Jul 01 19:54:51.590503 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:51.590478 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-b4182-serving-cert\"" Jul 01 19:54:51.590830 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:51.590813 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-b4182-kube-rbac-proxy-sar-config\"" Jul 01 19:54:51.597090 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:51.597047 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4"] Jul 01 19:54:51.600361 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:51.600340 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:54:51.689758 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:51.689728 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-openshift-service-ca-bundle\") pod \"splitter-graph-b4182-58cdcd74d5-w67h4\" (UID: \"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8\") " pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" Jul 01 19:54:51.689960 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:51.689786 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-proxy-tls\") pod \"splitter-graph-b4182-58cdcd74d5-w67h4\" (UID: \"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8\") " pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" Jul 01 19:54:51.689960 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:54:51.689905 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-b4182-serving-cert: secret "splitter-graph-b4182-serving-cert" not found Jul 01 19:54:51.690126 ip-10-0-138-252 kubenswrapper[2585]: E0701 19:54:51.689976 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-proxy-tls podName:90053a3e-e1d7-4367-9bcf-44f26a7c0ce8 nodeName:}" failed. No retries permitted until 2026-07-01 19:54:52.189953396 +0000 UTC m=+1899.736330007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-proxy-tls") pod "splitter-graph-b4182-58cdcd74d5-w67h4" (UID: "90053a3e-e1d7-4367-9bcf-44f26a7c0ce8") : secret "splitter-graph-b4182-serving-cert" not found Jul 01 19:54:51.690388 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:51.690368 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-openshift-service-ca-bundle\") pod \"splitter-graph-b4182-58cdcd74d5-w67h4\" (UID: \"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8\") " pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" Jul 01 19:54:52.062293 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:52.062257 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:54:52.193247 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:52.193210 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-proxy-tls\") pod \"splitter-graph-b4182-58cdcd74d5-w67h4\" (UID: \"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8\") " pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" Jul 01 19:54:52.195681 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:52.195649 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-proxy-tls\") pod \"splitter-graph-b4182-58cdcd74d5-w67h4\" (UID: \"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8\") " pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" Jul 01 19:54:52.197536 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:52.197517 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" Jul 01 19:54:52.333548 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:52.333484 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4"] Jul 01 19:54:52.335805 ip-10-0-138-252 kubenswrapper[2585]: W0701 19:54:52.335779 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90053a3e_e1d7_4367_9bcf_44f26a7c0ce8.slice/crio-691da111d956527d8cf10bafe1d29a366e54205495c53328f27909c42804ad83 WatchSource:0}: Error finding container 691da111d956527d8cf10bafe1d29a366e54205495c53328f27909c42804ad83: Status 404 returned error can't find the container with id 691da111d956527d8cf10bafe1d29a366e54205495c53328f27909c42804ad83 Jul 01 19:54:52.342485 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:52.342452 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" event={"ID":"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8","Type":"ContainerStarted","Data":"691da111d956527d8cf10bafe1d29a366e54205495c53328f27909c42804ad83"} Jul 01 19:54:53.348315 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:53.348271 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" event={"ID":"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8","Type":"ContainerStarted","Data":"d66c57875a8e5ad004ab839943f52472f9c222e3e3d87e38dc8461dda35558da"} Jul 01 19:54:53.348693 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:53.348345 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" Jul 01 19:54:53.365860 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:53.365808 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" podStartSLOduration=2.365790372 podStartE2EDuration="2.365790372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:54:53.364250721 +0000 UTC m=+1900.910627368" watchObservedRunningTime="2026-07-01 19:54:53.365790372 +0000 UTC m=+1900.912167007" Jul 01 19:54:59.357883 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:54:59.357847 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" Jul 01 19:57:30.781086 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:57:30.781033 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 19:58:13.033267 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:58:13.033235 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 19:58:13.037476 ip-10-0-138-252 kubenswrapper[2585]: I0701 19:58:13.037453 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 20:00:34.545739 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:00:34.545621 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:01:30.944115 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:01:30.944041 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:01:31.043900 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:01:31.043871 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:01:51.000027 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:01:50.999986 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:02:12.359021 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:02:12.358921 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:03:06.295973 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:06.295941 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4"] Jul 01 20:03:06.296600 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:06.296159 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" podUID="90053a3e-e1d7-4367-9bcf-44f26a7c0ce8" containerName="splitter-graph-b4182" containerID="cri-o://d66c57875a8e5ad004ab839943f52472f9c222e3e3d87e38dc8461dda35558da" gracePeriod=30 Jul 01 20:03:06.327245 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:06.327204 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:03:07.617161 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:07.617128 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:03:09.356199 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:09.356148 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" podUID="90053a3e-e1d7-4367-9bcf-44f26a7c0ce8" containerName="splitter-graph-b4182" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 20:03:13.058521 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:13.058493 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 20:03:13.063735 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:13.063712 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 20:03:14.357201 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:14.357153 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" podUID="90053a3e-e1d7-4367-9bcf-44f26a7c0ce8" containerName="splitter-graph-b4182" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 20:03:19.356207 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:19.356162 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" podUID="90053a3e-e1d7-4367-9bcf-44f26a7c0ce8" containerName="splitter-graph-b4182" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 20:03:19.356594 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:19.356261 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" Jul 01 20:03:24.356421 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:24.356378 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" podUID="90053a3e-e1d7-4367-9bcf-44f26a7c0ce8" containerName="splitter-graph-b4182" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 20:03:29.356719 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:29.356681 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" podUID="90053a3e-e1d7-4367-9bcf-44f26a7c0ce8" containerName="splitter-graph-b4182" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 20:03:34.356107 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:34.355996 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" podUID="90053a3e-e1d7-4367-9bcf-44f26a7c0ce8" containerName="splitter-graph-b4182" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 20:03:36.446436 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.446408 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" Jul 01 20:03:36.562587 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.562495 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-openshift-service-ca-bundle\") pod \"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8\" (UID: \"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8\") " Jul 01 20:03:36.562587 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.562531 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-proxy-tls\") pod \"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8\" (UID: \"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8\") " Jul 01 20:03:36.562903 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.562882 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "90053a3e-e1d7-4367-9bcf-44f26a7c0ce8" (UID: "90053a3e-e1d7-4367-9bcf-44f26a7c0ce8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 20:03:36.564783 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.564758 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "90053a3e-e1d7-4367-9bcf-44f26a7c0ce8" (UID: "90053a3e-e1d7-4367-9bcf-44f26a7c0ce8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 20:03:36.651901 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.651865 2585 generic.go:356] "Generic (PLEG): container finished" podID="90053a3e-e1d7-4367-9bcf-44f26a7c0ce8" containerID="d66c57875a8e5ad004ab839943f52472f9c222e3e3d87e38dc8461dda35558da" exitCode=0 Jul 01 20:03:36.652080 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.651911 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" Jul 01 20:03:36.652080 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.651950 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" event={"ID":"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8","Type":"ContainerDied","Data":"d66c57875a8e5ad004ab839943f52472f9c222e3e3d87e38dc8461dda35558da"} Jul 01 20:03:36.652080 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.652024 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4" event={"ID":"90053a3e-e1d7-4367-9bcf-44f26a7c0ce8","Type":"ContainerDied","Data":"691da111d956527d8cf10bafe1d29a366e54205495c53328f27909c42804ad83"} Jul 01 20:03:36.652080 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.652046 2585 scope.go:117] "RemoveContainer" containerID="d66c57875a8e5ad004ab839943f52472f9c222e3e3d87e38dc8461dda35558da" Jul 01 20:03:36.660541 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.660520 2585 scope.go:117] "RemoveContainer" containerID="d66c57875a8e5ad004ab839943f52472f9c222e3e3d87e38dc8461dda35558da" Jul 01 20:03:36.660803 ip-10-0-138-252 kubenswrapper[2585]: E0701 20:03:36.660781 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d66c57875a8e5ad004ab839943f52472f9c222e3e3d87e38dc8461dda35558da\": container with ID starting with d66c57875a8e5ad004ab839943f52472f9c222e3e3d87e38dc8461dda35558da not found: ID does not exist" containerID="d66c57875a8e5ad004ab839943f52472f9c222e3e3d87e38dc8461dda35558da" Jul 01 20:03:36.660868 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.660826 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66c57875a8e5ad004ab839943f52472f9c222e3e3d87e38dc8461dda35558da"} err="failed to get container status \"d66c57875a8e5ad004ab839943f52472f9c222e3e3d87e38dc8461dda35558da\": rpc error: code = NotFound desc = could not find container \"d66c57875a8e5ad004ab839943f52472f9c222e3e3d87e38dc8461dda35558da\": container with ID starting with d66c57875a8e5ad004ab839943f52472f9c222e3e3d87e38dc8461dda35558da not found: ID does not exist" Jul 01 20:03:36.663188 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.663168 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-openshift-service-ca-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 20:03:36.663188 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.663189 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 20:03:36.673030 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.673007 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4"] Jul 01 20:03:36.676859 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.676838 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b4182-58cdcd74d5-w67h4"] Jul 01 20:03:36.961336 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:03:36.961256 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90053a3e-e1d7-4367-9bcf-44f26a7c0ce8" path="/var/lib/kubelet/pods/90053a3e-e1d7-4367-9bcf-44f26a7c0ce8/volumes" Jul 01 20:04:21.965935 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:04:21.965896 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:08:13.083944 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:08:13.083871 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 20:08:13.089968 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:08:13.089947 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 20:08:40.926041 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:08:40.925996 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:11:00.627428 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:00.627389 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w"] Jul 01 20:11:00.627923 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:00.627634 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" podUID="fc929f5f-8d42-4360-9329-a4b81c206f7f" containerName="switch-graph-ee34f" containerID="cri-o://41aa0cba55edbc509ed34cb1e37108ef9fd1178aaef4028fc90603c015e41490" gracePeriod=30 Jul 01 20:11:00.710997 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:00.710967 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:11:00.815382 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:00.815346 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:11:00.880966 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:00.880848 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:11:01.759715 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:01.759675 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:11:01.849830 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:01.849796 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:11:03.306801 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:03.306758 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" podUID="fc929f5f-8d42-4360-9329-a4b81c206f7f" containerName="switch-graph-ee34f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 20:11:08.306319 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:08.306239 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" podUID="fc929f5f-8d42-4360-9329-a4b81c206f7f" containerName="switch-graph-ee34f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 20:11:13.306500 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:13.306463 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" podUID="fc929f5f-8d42-4360-9329-a4b81c206f7f" containerName="switch-graph-ee34f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 20:11:13.306877 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:13.306564 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" Jul 01 20:11:17.919143 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:17.919112 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ee34f-5765b8d4c6-r9x5w_fc929f5f-8d42-4360-9329-a4b81c206f7f/switch-graph-ee34f/0.log" Jul 01 20:11:18.306081 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:18.306029 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" podUID="fc929f5f-8d42-4360-9329-a4b81c206f7f" containerName="switch-graph-ee34f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 20:11:18.746851 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:18.746745 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ee34f-5765b8d4c6-r9x5w_fc929f5f-8d42-4360-9329-a4b81c206f7f/switch-graph-ee34f/0.log" Jul 01 20:11:19.571835 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:19.571797 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ee34f-5765b8d4c6-r9x5w_fc929f5f-8d42-4360-9329-a4b81c206f7f/switch-graph-ee34f/0.log" Jul 01 20:11:20.396478 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:20.396442 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ee34f-5765b8d4c6-r9x5w_fc929f5f-8d42-4360-9329-a4b81c206f7f/switch-graph-ee34f/0.log" Jul 01 20:11:21.234265 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:21.234232 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ee34f-5765b8d4c6-r9x5w_fc929f5f-8d42-4360-9329-a4b81c206f7f/switch-graph-ee34f/0.log" Jul 01 20:11:22.049721 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:22.049696 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ee34f-5765b8d4c6-r9x5w_fc929f5f-8d42-4360-9329-a4b81c206f7f/switch-graph-ee34f/0.log" Jul 01 20:11:22.874433 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:22.874401 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ee34f-5765b8d4c6-r9x5w_fc929f5f-8d42-4360-9329-a4b81c206f7f/switch-graph-ee34f/0.log" Jul 01 20:11:23.306378 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:23.306342 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" podUID="fc929f5f-8d42-4360-9329-a4b81c206f7f" containerName="switch-graph-ee34f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 20:11:23.686505 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:23.686431 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ee34f-5765b8d4c6-r9x5w_fc929f5f-8d42-4360-9329-a4b81c206f7f/switch-graph-ee34f/0.log" Jul 01 20:11:24.481905 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:24.481877 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ee34f-5765b8d4c6-r9x5w_fc929f5f-8d42-4360-9329-a4b81c206f7f/switch-graph-ee34f/0.log" Jul 01 20:11:25.304632 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:25.304603 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ee34f-5765b8d4c6-r9x5w_fc929f5f-8d42-4360-9329-a4b81c206f7f/switch-graph-ee34f/0.log" Jul 01 20:11:26.120964 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:26.120933 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ee34f-5765b8d4c6-r9x5w_fc929f5f-8d42-4360-9329-a4b81c206f7f/switch-graph-ee34f/0.log" Jul 01 20:11:26.940042 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:26.940012 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ee34f-5765b8d4c6-r9x5w_fc929f5f-8d42-4360-9329-a4b81c206f7f/switch-graph-ee34f/0.log" Jul 01 20:11:27.803358 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:27.803330 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ee34f-5765b8d4c6-r9x5w_fc929f5f-8d42-4360-9329-a4b81c206f7f/switch-graph-ee34f/0.log" Jul 01 20:11:28.305899 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:28.305860 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" podUID="fc929f5f-8d42-4360-9329-a4b81c206f7f" containerName="switch-graph-ee34f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 20:11:28.695290 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:28.695201 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ee34f-5765b8d4c6-r9x5w_fc929f5f-8d42-4360-9329-a4b81c206f7f/switch-graph-ee34f/0.log" Jul 01 20:11:30.715106 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:30.715043 2585 generic.go:356] "Generic (PLEG): container finished" podID="fc929f5f-8d42-4360-9329-a4b81c206f7f" containerID="41aa0cba55edbc509ed34cb1e37108ef9fd1178aaef4028fc90603c015e41490" exitCode=0 Jul 01 20:11:30.715512 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:30.715183 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" event={"ID":"fc929f5f-8d42-4360-9329-a4b81c206f7f","Type":"ContainerDied","Data":"41aa0cba55edbc509ed34cb1e37108ef9fd1178aaef4028fc90603c015e41490"} Jul 01 20:11:30.777289 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:30.777266 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" Jul 01 20:11:30.882768 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:30.882669 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc929f5f-8d42-4360-9329-a4b81c206f7f-openshift-service-ca-bundle\") pod \"fc929f5f-8d42-4360-9329-a4b81c206f7f\" (UID: \"fc929f5f-8d42-4360-9329-a4b81c206f7f\") " Jul 01 20:11:30.882768 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:30.882728 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc929f5f-8d42-4360-9329-a4b81c206f7f-proxy-tls\") pod \"fc929f5f-8d42-4360-9329-a4b81c206f7f\" (UID: \"fc929f5f-8d42-4360-9329-a4b81c206f7f\") " Jul 01 20:11:30.883115 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:30.883084 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc929f5f-8d42-4360-9329-a4b81c206f7f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fc929f5f-8d42-4360-9329-a4b81c206f7f" (UID: "fc929f5f-8d42-4360-9329-a4b81c206f7f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 20:11:30.884786 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:30.884758 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc929f5f-8d42-4360-9329-a4b81c206f7f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc929f5f-8d42-4360-9329-a4b81c206f7f" (UID: "fc929f5f-8d42-4360-9329-a4b81c206f7f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 20:11:30.984261 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:30.984236 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc929f5f-8d42-4360-9329-a4b81c206f7f-openshift-service-ca-bundle\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 20:11:30.984261 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:30.984262 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc929f5f-8d42-4360-9329-a4b81c206f7f-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\"" Jul 01 20:11:31.720511 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:31.720476 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" event={"ID":"fc929f5f-8d42-4360-9329-a4b81c206f7f","Type":"ContainerDied","Data":"384b960facdf4c0c4b6a22d06ae2aeedd1ffae63313ea5fee9064538a27014c4"} Jul 01 20:11:31.720511 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:31.720511 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w" Jul 01 20:11:31.721013 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:31.720546 2585 scope.go:117] "RemoveContainer" containerID="41aa0cba55edbc509ed34cb1e37108ef9fd1178aaef4028fc90603c015e41490" Jul 01 20:11:31.737999 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:31.737974 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w"] Jul 01 20:11:31.741930 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:31.741907 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ee34f-5765b8d4c6-r9x5w"] Jul 01 20:11:31.912187 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:31.912143 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:11:32.059284 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:32.059253 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:11:32.160324 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:32.160292 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:11:32.586118 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:32.586059 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:11:32.961586 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:32.961504 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc929f5f-8d42-4360-9329-a4b81c206f7f" path="/var/lib/kubelet/pods/fc929f5f-8d42-4360-9329-a4b81c206f7f/volumes" Jul 01 20:11:34.712216 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:34.712188 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qsrzd_e2553301-dbf4-4594-9dea-5d6f3b6ff8f2/global-pull-secret-syncer/0.log" Jul 01 20:11:34.753911 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:34.753884 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" Jul 01 20:11:34.892778 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:34.892750 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-252.ec2.internal_0ad6b0a84840892a426fb300e0c2d8d6/haproxy/0.log" Jul 01 20:11:37.645320 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:37.645282 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:11:37.746268 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:37.746215 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvn4g_2e4f997c-ae5b-4400-874a-89c94b3a5796/konnectivity-agent/0.log" limit=1024 Jul 01 20:11:39.056245 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:39.056213 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-54cb84584-gxrd5_ef688413-3e26-4dfd-87c9-059133228994/cluster-monitoring-operator/0.log" Jul 01 20:11:39.261951 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:39.261870 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dj7cs_cd873e60-27ca-4f25-a45a-54a66c46fc5a/node-exporter/0.log" Jul 01 20:11:39.284169 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:39.284142 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dj7cs_cd873e60-27ca-4f25-a45a-54a66c46fc5a/kube-rbac-proxy/0.log" Jul 01 20:11:39.306588 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:39.306559 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dj7cs_cd873e60-27ca-4f25-a45a-54a66c46fc5a/init-textfile/0.log" Jul 01 20:11:39.891750 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:39.891724 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7ddc8784d4-2njh6_10e4a2f9-1c4b-41b4-bb46-feb2922b4b26/prometheus-operator-admission-webhook/0.log" Jul 01 20:11:41.515381 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:41.515358 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-rgndr_9f78f69e-ffbb-4336-9976-daf5d2a9ea62/networking-console-plugin/0.log" Jul 01 20:11:42.044873 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:42.044842 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/2.log" Jul 01 20:11:42.048264 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:42.048246 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-m82rf_e53cdc86-b072-4c1e-9ca3-3d504c0bc6af/console-operator/3.log" Jul 01 20:11:42.501747 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:42.501671 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-594b45c979-x9rfm_a26f54f9-d4c9-424b-8871-afc6f762c801/console/0.log" Jul 01 20:11:43.055076 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.055048 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-69d9f679db-x7hw2_d93cdf36-ab87-4ac4-b4a3-5ed00053c479/volume-data-source-validator/0.log" Jul 01 20:11:43.304901 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.304864 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x"] Jul 01 20:11:43.305265 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.305222 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90053a3e-e1d7-4367-9bcf-44f26a7c0ce8" containerName="splitter-graph-b4182" Jul 01 20:11:43.305265 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.305237 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="90053a3e-e1d7-4367-9bcf-44f26a7c0ce8" containerName="splitter-graph-b4182" Jul 01 20:11:43.305265 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.305253 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc929f5f-8d42-4360-9329-a4b81c206f7f" containerName="switch-graph-ee34f" Jul 01 20:11:43.305265 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.305259 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc929f5f-8d42-4360-9329-a4b81c206f7f" containerName="switch-graph-ee34f" Jul 01 20:11:43.305413 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.305336 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc929f5f-8d42-4360-9329-a4b81c206f7f" containerName="switch-graph-ee34f" Jul 01 20:11:43.305413 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.305347 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="90053a3e-e1d7-4367-9bcf-44f26a7c0ce8" containerName="splitter-graph-b4182" Jul 01 20:11:43.309872 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.309852 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.312204 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.312185 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cn2r8\"/\"openshift-service-ca.crt\"" Jul 01 20:11:43.312352 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.312311 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cn2r8\"/\"default-dockercfg-f6k5p\"" Jul 01 20:11:43.312887 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.312871 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cn2r8\"/\"kube-root-ca.crt\"" Jul 01 20:11:43.321223 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.321203 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x"] Jul 01 20:11:43.380436 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.380403 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/782f5067-2439-4b23-b66e-3001c27b22c6-lib-modules\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.380436 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.380439 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/782f5067-2439-4b23-b66e-3001c27b22c6-podres\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.380667 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.380505 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/782f5067-2439-4b23-b66e-3001c27b22c6-sys\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.380667 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.380521 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjtc\" (UniqueName: \"kubernetes.io/projected/782f5067-2439-4b23-b66e-3001c27b22c6-kube-api-access-drjtc\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.380667 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.380586 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/782f5067-2439-4b23-b66e-3001c27b22c6-proc\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.481696 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.481665 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/782f5067-2439-4b23-b66e-3001c27b22c6-sys\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.481696 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.481700 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjtc\" (UniqueName: \"kubernetes.io/projected/782f5067-2439-4b23-b66e-3001c27b22c6-kube-api-access-drjtc\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.481874 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.481730 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/782f5067-2439-4b23-b66e-3001c27b22c6-proc\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.481874 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.481783 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/782f5067-2439-4b23-b66e-3001c27b22c6-lib-modules\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.481874 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.481788 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/782f5067-2439-4b23-b66e-3001c27b22c6-sys\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.481874 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.481806 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/782f5067-2439-4b23-b66e-3001c27b22c6-podres\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.481874 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.481852 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/782f5067-2439-4b23-b66e-3001c27b22c6-proc\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.482046 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.481902 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/782f5067-2439-4b23-b66e-3001c27b22c6-podres\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.482046 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.481916 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/782f5067-2439-4b23-b66e-3001c27b22c6-lib-modules\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.490141 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.490116 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjtc\" (UniqueName: \"kubernetes.io/projected/782f5067-2439-4b23-b66e-3001c27b22c6-kube-api-access-drjtc\") pod \"perf-node-gather-daemonset-kzw6x\" (UID: \"782f5067-2439-4b23-b66e-3001c27b22c6\") " pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.619111 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.619003 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:43.752596 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.752569 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x"] Jul 01 20:11:43.755275 ip-10-0-138-252 kubenswrapper[2585]: W0701 20:11:43.755236 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod782f5067_2439_4b23_b66e_3001c27b22c6.slice/crio-636461d1151900282558dcbaba5eacaecd14ee008088c0906d43fb9640003368 WatchSource:0}: Error finding container 636461d1151900282558dcbaba5eacaecd14ee008088c0906d43fb9640003368: Status 404 returned error can't find the container with id 636461d1151900282558dcbaba5eacaecd14ee008088c0906d43fb9640003368 Jul 01 20:11:43.774086 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:43.774041 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" event={"ID":"782f5067-2439-4b23-b66e-3001c27b22c6","Type":"ContainerStarted","Data":"636461d1151900282558dcbaba5eacaecd14ee008088c0906d43fb9640003368"} Jul 01 20:11:44.015292 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:44.015219 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j8kx5_9f285b32-ab0f-4888-bdc3-8c9f84a535cd/dns/0.log" Jul 01 20:11:44.037428 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:44.037407 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j8kx5_9f285b32-ab0f-4888-bdc3-8c9f84a535cd/kube-rbac-proxy/0.log" Jul 01 20:11:44.169740 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:44.169715 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cgn6q_a40a5624-aef1-4d71-a765-ccc60715a3d8/dns-node-resolver/0.log" Jul 01 20:11:44.735075 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:44.735036 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-926nd_04296533-7323-4246-bcf2-fc3cd1e387ef/node-ca/0.log" Jul 01 20:11:44.780084 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:44.780034 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" event={"ID":"782f5067-2439-4b23-b66e-3001c27b22c6","Type":"ContainerStarted","Data":"899f5fcf6e3c23145e7d56948c77f2492efc0147a1be82f2e3950416bb3b3a3f"} Jul 01 20:11:44.780246 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:44.780104 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:44.797763 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:44.797709 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" podStartSLOduration=1.7976960480000002 podStartE2EDuration="1.797696048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:11:44.796151886 +0000 UTC m=+2912.342528524" watchObservedRunningTime="2026-07-01 20:11:44.797696048 +0000 UTC m=+2912.344072678" Jul 01 20:11:45.606590 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:45.606565 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fb847847c-xqccn_0c94f978-1520-4f79-91df-f1f0d8048877/router/0.log" Jul 01 20:11:46.051135 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:46.051104 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z82nd_952e2ea8-73dc-4456-9ae4-29fc1e2cdb02/serve-healthcheck-canary/0.log" Jul 01 20:11:46.416326 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:46.416253 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-428zv_8f60a799-ece9-40f1-a3ff-44f5f10ef0a0/insights-operator/0.log" Jul 01 20:11:46.416786 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:46.416772 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-428zv_8f60a799-ece9-40f1-a3ff-44f5f10ef0a0/insights-operator/1.log" Jul 01 20:11:46.580943 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:46.580914 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdm9z_0510fe39-50bf-472c-adc1-f078f5406fdd/kube-rbac-proxy/0.log" Jul 01 20:11:46.602962 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:46.602935 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdm9z_0510fe39-50bf-472c-adc1-f078f5406fdd/exporter/0.log" Jul 01 20:11:46.627367 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:46.627344 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdm9z_0510fe39-50bf-472c-adc1-f078f5406fdd/extractor/0.log" Jul 01 20:11:48.760469 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:48.760439 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-79d94fb575-xzgv5_e0f673c1-84cc-490e-8c86-58a4be42ce67/manager/0.log" Jul 01 20:11:48.782448 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:48.782420 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-n5k4k_0fae09df-5ec4-4719-a8e9-7deb4bcc56ca/manager/0.log" Jul 01 20:11:50.794053 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:50.794025 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cn2r8/perf-node-gather-daemonset-kzw6x" Jul 01 20:11:52.983359 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:52.983331 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-nnfgg_9070bac0-2f2a-4fc2-b184-7363d8aa12ac/migrator/0.log" Jul 01 20:11:53.005646 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:53.005624 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-nnfgg_9070bac0-2f2a-4fc2-b184-7363d8aa12ac/graceful-termination/0.log" Jul 01 20:11:53.457337 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:53.457259 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-685pf_5947ec3f-7e65-49cf-b511-44dc3b526ff2/kube-storage-version-migrator-operator/1.log" Jul 01 20:11:53.458135 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:53.458115 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-685pf_5947ec3f-7e65-49cf-b511-44dc3b526ff2/kube-storage-version-migrator-operator/0.log" Jul 01 20:11:54.494848 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:54.494821 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-plcl5_90fc2dc7-c463-49e5-b2f5-050c9238f37c/kube-multus-additional-cni-plugins/0.log" Jul 01 20:11:54.518830 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:54.518802 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-plcl5_90fc2dc7-c463-49e5-b2f5-050c9238f37c/egress-router-binary-copy/0.log" Jul 01 20:11:54.545420 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:54.545388 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-plcl5_90fc2dc7-c463-49e5-b2f5-050c9238f37c/cni-plugins/0.log" Jul 01 20:11:54.573072 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:54.573037 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-plcl5_90fc2dc7-c463-49e5-b2f5-050c9238f37c/bond-cni-plugin/0.log" Jul 01 20:11:54.599591 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:54.599565 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-plcl5_90fc2dc7-c463-49e5-b2f5-050c9238f37c/routeoverride-cni/0.log" Jul 01 20:11:54.629424 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:54.629400 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-plcl5_90fc2dc7-c463-49e5-b2f5-050c9238f37c/whereabouts-cni-bincopy/0.log" Jul 01 20:11:54.654442 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:54.654413 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-plcl5_90fc2dc7-c463-49e5-b2f5-050c9238f37c/whereabouts-cni/0.log" Jul 01 20:11:55.109601 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:55.109515 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zjd48_e658cc5b-e0b2-4701-8568-5008df00ed5b/kube-multus/0.log" Jul 01 20:11:55.185727 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:55.185696 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m2flh_688cb495-bdbd-40fc-a726-7bc07d269276/network-metrics-daemon/0.log" Jul 01 20:11:55.207936 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:55.207911 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m2flh_688cb495-bdbd-40fc-a726-7bc07d269276/kube-rbac-proxy/0.log" Jul 01 20:11:56.239773 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:56.239738 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-frt9f_8c7695a8-08e8-4dc3-8c0f-121f69a158a4/ovn-controller/0.log" Jul 01 20:11:56.261044 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:56.261024 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-frt9f_8c7695a8-08e8-4dc3-8c0f-121f69a158a4/ovn-acl-logging/0.log" Jul 01 20:11:56.288845 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:56.288819 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-frt9f_8c7695a8-08e8-4dc3-8c0f-121f69a158a4/kube-rbac-proxy-node/0.log" Jul 01 20:11:56.311610 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:56.311589 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-frt9f_8c7695a8-08e8-4dc3-8c0f-121f69a158a4/kube-rbac-proxy-ovn-metrics/0.log" Jul 01 20:11:56.333835 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:56.333814 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-frt9f_8c7695a8-08e8-4dc3-8c0f-121f69a158a4/northd/0.log" Jul 01 20:11:56.358333 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:56.358316 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-frt9f_8c7695a8-08e8-4dc3-8c0f-121f69a158a4/nbdb/0.log" Jul 01 20:11:56.383261 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:56.383237 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-frt9f_8c7695a8-08e8-4dc3-8c0f-121f69a158a4/sbdb/0.log" Jul 01 20:11:56.427318 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:56.427290 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-frt9f_8c7695a8-08e8-4dc3-8c0f-121f69a158a4/ovnkube-controller/0.log" Jul 01 20:11:57.674463 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:57.674439 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-56499946bc-sn7px_19decc52-62ea-4657-a5e7-dafa4b580eb8/check-endpoints/0.log" Jul 01 20:11:57.725756 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:57.725726 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-csgsb_54990c60-95a2-4a2c-9d4d-690c9994707b/network-check-target-container/0.log" Jul 01 20:11:58.671333 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:58.671306 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-d6rls_fba7d267-ad49-4729-8e5f-737cf617fcd6/iptables-alerter/0.log" Jul 01 20:11:59.324738 ip-10-0-138-252 kubenswrapper[2585]: I0701 20:11:59.324660 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-b9f2f_7ce6745e-2896-41de-b042-33adaac95142/tuned/0.log"