Jul 06 23:54:20.471318 ip-10-0-129-125 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 23:54:20.471332 ip-10-0-129-125 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 23:54:20.471341 ip-10-0-129-125 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 23:54:20.471724 ip-10-0-129-125 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 23:54:29.935653 ip-10-0-129-125 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 06 23:54:29.935671 ip-10-0-129-125 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c8e4297aad3f43ea930b55a63f581742 -- Jul 06 23:56:16.728704 ip-10-0-129-125 systemd[1]: Starting Kubernetes Kubelet... Jul 06 23:56:17.222090 ip-10-0-129-125 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:56:17.222090 ip-10-0-129-125 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 23:56:17.222090 ip-10-0-129-125 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:56:17.222090 ip-10-0-129-125 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 23:56:17.222090 ip-10-0-129-125 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:56:17.222090 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.219163 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225854 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225864 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225868 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225872 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225875 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225878 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225881 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225883 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225887 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225890 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225892 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225895 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225897 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225900 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225903 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225905 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225908 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225911 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225913 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:17.226013 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225917 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225919 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225922 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225925 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225927 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225930 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225932 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225936 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225939 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225941 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225944 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225946 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225949 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225951 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225954 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225957 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225959 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225962 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225964 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225967 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:17.340882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225969 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:17.316146 ip-10-0-129-125 systemd[1]: Started Kubernetes Kubelet. Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225972 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225974 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225977 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225980 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225983 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225987 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225989 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225993 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.225997 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226000 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226003 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226006 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226010 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226012 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226015 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226018 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226021 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226024 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:17.341670 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226029 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226033 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226036 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226039 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226041 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226044 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226046 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226049 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226051 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226054 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226057 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226060 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226063 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226065 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226069 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226071 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226074 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226077 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226080 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:17.342284 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226083 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226087 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226090 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226093 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226096 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226098 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226102 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226105 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226107 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226110 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226113 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226116 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226119 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226122 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226124 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226127 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226130 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226132 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226135 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226137 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:17.342923 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226140 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226142 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226846 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226853 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226856 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226859 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226862 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226864 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226867 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226871 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226875 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226878 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226881 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226884 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226887 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226890 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226893 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226895 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226898 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:17.343529 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226901 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226903 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226906 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226908 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226911 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226913 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226916 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226921 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226925 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226928 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226931 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226933 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226937 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226939 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226943 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226945 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226948 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226951 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226953 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:17.344137 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226956 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226959 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226961 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226964 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226966 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226969 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226971 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226974 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226976 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226979 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226982 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226984 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226987 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226989 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226992 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226994 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226997 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.226999 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227001 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227004 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:17.344659 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227006 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227009 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227012 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227014 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227016 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227019 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227022 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227024 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227027 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227030 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227033 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227035 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227039 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227042 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227045 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227048 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227051 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227054 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227056 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:17.345405 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227059 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227061 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227065 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227068 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227071 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227073 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227075 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227078 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227080 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227083 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227085 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227088 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227090 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227093 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227095 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227098 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227100 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227103 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227106 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227110 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:17.346151 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227113 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227116 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227118 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227121 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227492 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227497 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227499 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227502 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227506 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227510 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227513 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227516 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227519 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227521 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227524 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227529 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227532 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227536 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227538 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227541 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:17.346777 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227544 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227547 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227549 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227552 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227554 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227557 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227559 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227562 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227565 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227568 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227570 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227575 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227577 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227580 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227583 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227586 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227588 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227591 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227593 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:17.347289 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227596 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227599 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227601 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227604 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227606 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227608 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227611 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227614 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227616 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227619 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227622 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227625 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227628 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227631 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227634 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227636 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227639 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227641 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227644 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227646 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:17.347864 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227649 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227652 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227655 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227657 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227661 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227663 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227666 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227668 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227671 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227673 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227676 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227678 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227681 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227683 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227686 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227688 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227691 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227693 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227696 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227698 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:17.348392 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227701 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227703 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227706 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227710 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227714 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227717 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227720 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227725 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227728 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227730 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227733 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227736 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227740 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227743 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227747 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227750 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227754 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227757 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227760 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:17.351282 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227763 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227765 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227768 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227771 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.227774 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227837 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227844 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227849 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227853 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227857 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227861 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227864 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227868 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227871 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227874 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227878 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227882 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227885 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227888 2568 flags.go:64] FLAG: --cgroup-root="" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227891 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227894 2568 flags.go:64] FLAG: --client-ca-file="" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227897 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227900 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227903 2568 flags.go:64] FLAG: --cluster-domain="" Jul 06 23:56:17.352069 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227906 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227910 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227913 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227917 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227920 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227925 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227928 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227931 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227934 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227937 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227940 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227943 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227946 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227950 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227953 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227955 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227958 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227961 2568 flags.go:64] FLAG: --enable-server="true" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227964 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227967 2568 flags.go:64] FLAG: --event-burst="100" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227970 2568 flags.go:64] FLAG: --event-qps="50" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227973 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227976 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227979 2568 flags.go:64] FLAG: --eviction-hard="" Jul 06 23:56:17.353099 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227982 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227985 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227988 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227991 2568 flags.go:64] FLAG: --eviction-soft="" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227994 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227997 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.227999 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228002 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228005 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228009 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228012 2568 flags.go:64] FLAG: --feature-gates="" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228015 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228018 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228022 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228025 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228028 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228031 2568 flags.go:64] FLAG: --help="false" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228034 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228037 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228039 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228042 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228046 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228049 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228052 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 23:56:17.353918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228054 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228057 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228060 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228063 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228066 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228068 2568 flags.go:64] FLAG: --kube-reserved="" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228071 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228074 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228077 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228080 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228083 2568 flags.go:64] FLAG: --lock-file="" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228085 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228088 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228091 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228096 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228099 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228101 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228106 2568 flags.go:64] FLAG: --logging-format="text" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228108 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228112 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228114 2568 flags.go:64] FLAG: --manifest-url="" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228118 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228122 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228125 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228129 2568 flags.go:64] FLAG: --max-pods="110" Jul 06 23:56:17.354714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228132 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228135 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228137 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228140 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228143 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228146 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228149 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228155 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228158 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228161 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228164 2568 flags.go:64] FLAG: --pod-cidr="" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228167 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228172 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228174 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228177 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228180 2568 flags.go:64] FLAG: --port="10250" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228184 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228187 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-034615122a586304c" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228190 2568 flags.go:64] FLAG: --qos-reserved="" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228193 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228196 2568 flags.go:64] FLAG: --register-node="true" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228199 2568 flags.go:64] FLAG: --register-with-taints="" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228202 2568 flags.go:64] FLAG: --registry-burst="10" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228205 2568 flags.go:64] FLAG: --registry-qps="5" Jul 06 23:56:17.355350 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228209 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228212 2568 flags.go:64] FLAG: --reserved-memory="" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228215 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228218 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228221 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228225 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228228 2568 flags.go:64] FLAG: --runonce="false" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228231 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228234 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228237 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228239 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228242 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228245 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228248 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228251 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228254 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228257 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228260 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228262 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228265 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228268 2568 flags.go:64] FLAG: --system-cgroups="" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228271 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228276 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228279 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228282 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 23:56:17.356074 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228285 2568 flags.go:64] FLAG: --tls-min-version="" Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228288 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228291 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228294 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228297 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228300 2568 flags.go:64] FLAG: --v="2" Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228303 2568 flags.go:64] FLAG: --version="false" Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228308 2568 flags.go:64] FLAG: --vmodule="" Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228311 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228315 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228452 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228456 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228461 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228464 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228466 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228469 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228473 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228475 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228478 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228481 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228483 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228486 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:17.357271 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228489 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228491 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228494 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228496 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228499 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228502 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228504 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228507 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228509 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228512 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228514 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228517 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228520 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228523 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228526 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228529 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228531 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228535 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228537 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228540 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:17.358039 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228543 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228547 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228551 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228554 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228557 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228560 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228562 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228566 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228570 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228573 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228576 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228579 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228582 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228584 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228587 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228590 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228592 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228595 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228597 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:17.484115 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228600 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228603 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228605 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228608 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228610 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228613 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228616 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228618 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228621 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228623 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228627 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228630 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228632 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228635 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228637 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228641 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228644 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228647 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228649 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228652 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:17.526012 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228655 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228657 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228660 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228663 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228665 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228668 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228670 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228673 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228676 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228679 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228681 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228684 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228686 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228689 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228691 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228694 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228697 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228699 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228702 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228704 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:17.526734 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228707 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:17.529811 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228710 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:17.529811 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228713 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:17.529811 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228717 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:17.529811 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228721 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:17.529811 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228724 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:17.529811 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228726 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:17.529811 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.228731 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:17.529811 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.228736 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:56:17.529811 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.236644 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 23:56:17.529811 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.236658 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 23:56:17.529811 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.238026 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 23:56:17.529811 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.238037 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 23:56:17.529811 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238111 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:17.529811 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238116 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238121 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238126 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238130 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238135 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238139 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238143 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238147 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238152 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238156 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238160 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238164 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238168 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238172 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238176 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238179 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238183 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238187 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238191 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238196 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:17.530449 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238200 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238205 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238209 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238215 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238220 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238224 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238228 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238231 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238236 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238242 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238246 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238250 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238254 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238257 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238261 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238266 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238271 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238275 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238279 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238283 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:17.531354 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238287 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238291 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238294 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238298 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238302 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238307 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238311 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238314 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238318 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238322 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238326 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238329 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238332 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238335 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238339 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238343 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238346 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238350 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238354 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238358 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:17.532224 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238361 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238367 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238371 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238375 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238378 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238382 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238386 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238390 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238396 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238400 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238404 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238408 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238412 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238430 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238435 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238439 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238443 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238446 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238452 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238455 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:17.532899 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238459 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238463 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238467 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238471 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238475 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238478 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238482 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238486 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238489 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238493 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238497 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238500 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238504 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238510 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238516 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238521 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238525 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:17.533487 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238531 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:17.534120 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.238540 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:56:17.534120 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238718 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:17.534120 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238725 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:17.534120 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238731 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:17.534120 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238736 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:17.534120 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238741 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:17.534120 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238746 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:17.534120 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238751 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:17.534120 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238754 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:17.534120 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238758 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:17.534120 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238762 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:17.534120 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238765 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:17.534120 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238768 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:17.534120 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238772 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238774 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238777 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238779 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238782 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238784 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238787 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238789 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238792 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238795 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238797 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238800 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238802 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238806 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238810 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238813 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238816 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238818 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238821 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238823 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:17.534590 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238826 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238829 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238833 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238835 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238838 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238840 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238843 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238845 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238848 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238850 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238853 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238855 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238858 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238861 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238863 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238866 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238869 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238871 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238874 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238876 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:17.535246 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238879 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238881 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238884 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238886 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238889 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238891 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238895 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238898 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238900 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238903 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238905 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238908 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238910 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238913 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238918 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238920 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238923 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238925 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238929 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:17.535873 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238933 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238936 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238939 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238941 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238944 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238947 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238950 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238954 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238956 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238959 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238962 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238964 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238967 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238969 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238972 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238974 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238977 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238980 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238982 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238986 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:17.536474 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238989 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:17.537107 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238991 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:17.537107 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238993 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:17.537107 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238996 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:17.537107 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.238998 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:17.537107 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.239001 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:17.537107 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.239004 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:17.537107 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:17.239007 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:17.537107 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.239012 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:56:17.537107 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.239102 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 23:56:17.537107 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.242928 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 23:56:17.537107 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.244884 2568 server.go:1019] "Starting client certificate rotation" Jul 06 23:56:17.537107 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.244985 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 23:56:17.537107 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.245015 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 23:56:17.537561 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.248437 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 23:56:17.537561 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.251763 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 23:56:17.537561 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.272246 2568 log.go:25] "Validated CRI v1 runtime API" Jul 06 23:56:17.537561 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.274302 2568 log.go:25] "Validated CRI v1 image API" Jul 06 23:56:17.537561 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.275739 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 23:56:17.537561 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.278526 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 23:56:17.537561 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.278586 2568 fs.go:135] Filesystem UUIDs: map[614fea7f-57e7-49ed-ba01-d9dbceba8346:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ab50c901-0db4-4789-bcfb-387730046463:/dev/nvme0n1p4] Jul 06 23:56:17.537561 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.278625 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.287101 2568 manager.go:216] Machine: {Timestamp:2026-07-06 23:56:17.284619362 +0000 UTC m=+0.425076408 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099689 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec215252d5a74e22e036058c21880c47 SystemUUID:ec215252-d5a7-4e22-e036-058c21880c47 BootID:c8e4297a-ad3f-43ea-930b-55a63f581742 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a1:d2:a6:07:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a1:d2:a6:07:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:cd:08:ae:7f:f5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.287194 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.287265 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.290027 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.290047 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-125.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.290740 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.290750 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.290758 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.292398 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.293812 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.298088 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.298216 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.298245 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.298258 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.298272 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.298282 2568 kubelet.go:398] "Adding apiserver pod source" Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.298290 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 23:56:17.537918 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.298622 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.298637 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.301166 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tcpb2" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.305682 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.307601 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.307627 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.308889 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.308907 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.308916 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.308923 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.308937 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.308946 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.308956 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.308965 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.308975 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.308983 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.308992 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.308997 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.309862 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tcpb2" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.310503 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.310517 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.311945 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.311958 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.313732 2568 server.go:1268] "Started kubelet" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.313872 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.313925 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.314301 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-125.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.314486 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.316006 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.317477 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.321588 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.322988 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.323036 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.323058 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.323895 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.323909 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.323822 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.324059 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.324066 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.325054 2568 factory.go:55] Registering systemd factory Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.325071 2568 factory.go:223] Registration of the systemd container factory successfully Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.325291 2568 factory.go:153] Registering CRI-O factory Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.325307 2568 factory.go:223] Registration of the crio container factory successfully Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.325358 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.325386 2568 factory.go:103] Registering Raw factory Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.325400 2568 manager.go:1196] Started watching for new ooms in manager Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.325826 2568 manager.go:349] Starting recovery of all containers Jul 06 23:56:17.538602 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.326821 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.328801 2568 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-125.ec2.internal\" not found" node="ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.331869 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.337928 2568 manager.go:354] Recovery completed Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.342782 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.344656 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.344681 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.344690 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.345197 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.345205 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.345220 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.345269 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.345929 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.345958 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.345973 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.347354 2568 policy_none.go:49] "None policy: Start" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.347366 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.347376 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.349352 2568 policy_none.go:47] "Start" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.395921 2568 manager.go:337] "Starting Device Plugin manager" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.396040 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.396051 2568 server.go:87] "Starting device plugin registration server" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.396256 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.396265 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.396341 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.396408 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.396436 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.397007 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.397055 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-125.ec2.internal\" not found" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.479968 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.480002 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.480021 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.480050 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.482483 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.505896 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.506709 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.506736 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.506747 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.506768 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.516677 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.516694 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-125.ec2.internal\": node \"ip-10-0-129-125.ec2.internal\" not found" Jul 06 23:56:17.540004 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.535247 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.582500 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"] Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.582566 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.585530 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.585555 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.585569 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.587948 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.588088 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.588112 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.588695 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.588707 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.588718 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.588729 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.588731 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.588745 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.591567 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.591590 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.592236 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.592259 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.592273 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.625214 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2fa7c948617b9fac2532d316a93a8386-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.ec2.internal\" (UID: \"2fa7c948617b9fac2532d316a93a8386\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.625234 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1874cbf2171f4c31314a83eeadd2b7f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"1874cbf2171f4c31314a83eeadd2b7f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.625252 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1874cbf2171f4c31314a83eeadd2b7f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"1874cbf2171f4c31314a83eeadd2b7f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.625257 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-125.ec2.internal\" not found" node="ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.629883 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-125.ec2.internal\" not found" node="ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.722744 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.635973 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found" Jul 06 23:56:17.931180 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.726328 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2fa7c948617b9fac2532d316a93a8386-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.ec2.internal\" (UID: \"2fa7c948617b9fac2532d316a93a8386\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.931180 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.726349 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1874cbf2171f4c31314a83eeadd2b7f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"1874cbf2171f4c31314a83eeadd2b7f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.931180 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.726367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1874cbf2171f4c31314a83eeadd2b7f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"1874cbf2171f4c31314a83eeadd2b7f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.931180 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.726405 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1874cbf2171f4c31314a83eeadd2b7f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"1874cbf2171f4c31314a83eeadd2b7f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.931180 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.726459 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1874cbf2171f4c31314a83eeadd2b7f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"1874cbf2171f4c31314a83eeadd2b7f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.931180 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.726459 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2fa7c948617b9fac2532d316a93a8386-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.ec2.internal\" (UID: \"2fa7c948617b9fac2532d316a93a8386\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" Jul 06 23:56:17.931180 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.737050 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found" Jul 06 23:56:17.931180 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.837774 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found" Jul 06 23:56:17.931180 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.930025 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" Jul 06 23:56:18.071560 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:17.933206 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" Jul 06 23:56:18.071560 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:17.937939 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found" Jul 06 23:56:18.071560 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:18.038281 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found" Jul 06 23:56:18.197473 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:18.138876 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found" Jul 06 23:56:18.320098 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:18.239583 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found" Jul 06 23:56:18.320098 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:18.243766 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 23:56:18.320098 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:18.243941 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 23:56:18.320098 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:18.243953 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 23:56:18.320098 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:18.314449 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 23:51:18 +0000 UTC" deadline="2028-03-26 15:11:29.68885517 +0000 UTC" Jul 06 23:56:18.320098 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:18.314502 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15087h15m11.374358051s" Jul 06 23:56:18.340339 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:18.339736 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found" Jul 06 23:56:18.451966 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:18.422539 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:18.451966 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:18.423555 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" Jul 06 23:56:18.451966 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:18.434207 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 23:56:18.451966 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:18.436281 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" Jul 06 23:56:18.451966 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:18.446570 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 23:56:18.850191 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:18.758925 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:19.299373 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.299345 2568 apiserver.go:52] "Watching apiserver" Jul 06 23:56:19.305930 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.305908 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 23:56:19.306303 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.306281 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-4t5gn","openshift-multus/multus-additional-cni-plugins-qlz2q","openshift-network-diagnostics/network-check-target-bcn92","openshift-ovn-kubernetes/ovnkube-node-rg8hd","kube-system/global-pull-secret-syncer-x5k59","openshift-cluster-node-tuning-operator/tuned-2vpk2","openshift-dns/node-resolver-s8cmz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal","openshift-multus/multus-fnfp4","openshift-multus/network-metrics-daemon-mddqt","openshift-network-operator/iptables-alerter-8t6pk","kube-system/konnectivity-agent-tdb62","kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w"] Jul 06 23:56:19.309886 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.309865 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s8cmz" Jul 06 23:56:19.312330 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.312303 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8t6pk" Jul 06 23:56:19.312521 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.312422 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 23:56:19.312521 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.312424 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 23:56:19.312901 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.312882 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g7jsw\"" Jul 06 23:56:19.315314 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.315295 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 23:56:19.315593 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.315576 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:19.315686 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.315626 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:56:19.315686 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.315628 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vl48l\"" Jul 06 23:56:19.315686 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.315652 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 23:56:19.315686 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:19.315650 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:19.318286 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.318267 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.320558 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.320515 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 23:56:19.320908 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.320636 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 23:56:19.320908 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.320866 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 23:56:19.320908 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.320901 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 23:56:19.321055 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.320901 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xchlh\"" Jul 06 23:56:19.321055 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.320943 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 23:56:19.321183 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.321169 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 23:56:19.322524 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.321640 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:19.322524 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:19.321728 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:19.323462 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.323441 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 23:56:19.324445 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.324412 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.326597 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.326578 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qt4c8\"" Jul 06 23:56:19.326738 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.326720 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:56:19.327100 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.327085 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 23:56:19.327654 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.327634 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.330405 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.330379 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 23:56:19.330405 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.330395 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 23:56:19.330405 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.330405 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 23:56:19.330679 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.330420 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 23:56:19.330679 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.330478 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rn9p4\"" Jul 06 23:56:19.330679 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.330507 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.332644 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.332627 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 23:56:19.332744 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.332728 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 23:56:19.333281 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.333266 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.333387 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.333370 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 23:56:19.333578 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.333553 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xcf5v\"" Jul 06 23:56:19.335276 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335257 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3979fa5-6b05-4e90-abf0-ae3cb0286f0c-hosts-file\") pod \"node-resolver-s8cmz\" (UID: \"e3979fa5-6b05-4e90-abf0-ae3cb0286f0c\") " pod="openshift-dns/node-resolver-s8cmz" Jul 06 23:56:19.335339 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335287 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9d35a3e-274a-4e04-84f5-176b933aa822-host-slash\") pod \"iptables-alerter-8t6pk\" (UID: \"c9d35a3e-274a-4e04-84f5-176b933aa822\") " pod="openshift-network-operator/iptables-alerter-8t6pk" Jul 06 23:56:19.335339 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335317 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k55bq\" (UniqueName: \"kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq\") pod \"network-check-target-bcn92\" (UID: \"388db162-a514-4da6-b4ad-398a94e22e01\") " pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:19.335339 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-sysctl-d\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.335475 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335348 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-sysctl-conf\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.335475 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335363 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-sys\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.335475 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335379 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-lib-modules\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.335475 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335405 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-cni-bin\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.335475 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335430 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpgdk\" (UniqueName: \"kubernetes.io/projected/90898b8a-6bbd-45ae-b92b-24033e583d13-kube-api-access-jpgdk\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.335735 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335479 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:19.335735 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335518 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-96dd5\"" Jul 06 23:56:19.335735 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335528 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-sysconfig\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.335735 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335536 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 23:56:19.335735 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335532 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 23:56:19.335735 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335555 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2662ed3-dc3a-402d-bce7-0657390f9350-tmp\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.335735 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335581 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-systemd-units\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.335735 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335606 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-run-openvswitch\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.335735 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335528 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 23:56:19.335735 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335629 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90898b8a-6bbd-45ae-b92b-24033e583d13-ovnkube-config\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.335735 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335690 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90898b8a-6bbd-45ae-b92b-24033e583d13-ovn-node-metrics-cert\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.335735 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335708 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-run\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.335735 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335724 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-var-lib-kubelet\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-host\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335795 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-etc-openvswitch\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335831 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-run-ovn\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335854 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-kubelet\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335872 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3979fa5-6b05-4e90-abf0-ae3cb0286f0c-tmp-dir\") pod \"node-resolver-s8cmz\" (UID: \"e3979fa5-6b05-4e90-abf0-ae3cb0286f0c\") " pod="openshift-dns/node-resolver-s8cmz" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335897 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-run-systemd\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335927 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-systemd\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335945 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.335960 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-tuned\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:19.335993 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336002 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql99h\" (UniqueName: \"kubernetes.io/projected/e3979fa5-6b05-4e90-abf0-ae3cb0286f0c-kube-api-access-ql99h\") pod \"node-resolver-s8cmz\" (UID: \"e3979fa5-6b05-4e90-abf0-ae3cb0286f0c\") " pod="openshift-dns/node-resolver-s8cmz" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336027 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9d35a3e-274a-4e04-84f5-176b933aa822-iptables-alerter-script\") pod \"iptables-alerter-8t6pk\" (UID: \"c9d35a3e-274a-4e04-84f5-176b933aa822\") " pod="openshift-network-operator/iptables-alerter-8t6pk" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336049 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-slash\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336075 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-run-netns\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336098 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-run-ovn-kubernetes\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336121 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76bea87f-a057-4a59-8e4a-171542fa018a-kubelet-config\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:19.336185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336148 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-kubernetes\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.336751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336174 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr9ss\" (UniqueName: \"kubernetes.io/projected/c9d35a3e-274a-4e04-84f5-176b933aa822-kube-api-access-lr9ss\") pod \"iptables-alerter-8t6pk\" (UID: \"c9d35a3e-274a-4e04-84f5-176b933aa822\") " pod="openshift-network-operator/iptables-alerter-8t6pk" Jul 06 23:56:19.336751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336201 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-log-socket\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.336751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336246 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90898b8a-6bbd-45ae-b92b-24033e583d13-ovnkube-script-lib\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.336751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336269 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr7nj\" (UniqueName: \"kubernetes.io/projected/f2662ed3-dc3a-402d-bce7-0657390f9350-kube-api-access-tr7nj\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.336751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336299 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-var-lib-openvswitch\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.336751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336331 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-node-log\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.336751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336353 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-cni-netd\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.336751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336371 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.336751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336387 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76bea87f-a057-4a59-8e4a-171542fa018a-dbus\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:19.336751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336402 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-modprobe-d\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.336751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.336458 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90898b8a-6bbd-45ae-b92b-24033e583d13-env-overrides\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.339262 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.339243 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4t5gn" Jul 06 23:56:19.341262 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.341239 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 23:56:19.341560 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.341541 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 23:56:19.341663 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.341617 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gxhrf\"" Jul 06 23:56:19.341860 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.341809 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tdb62" Jul 06 23:56:19.342312 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.342289 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 23:56:19.343966 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.343951 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-thv5f\"" Jul 06 23:56:19.344080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.344038 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 23:56:19.344080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.344068 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 23:56:19.361029 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.360986 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xhgmq" Jul 06 23:56:19.369579 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.369561 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xhgmq" Jul 06 23:56:19.425185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.425161 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 23:56:19.437552 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437532 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.437649 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437558 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs\") pod \"network-metrics-daemon-mddqt\" (UID: \"b8d4e21f-d7f0-404d-a8e8-4da642d240c7\") " pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:19.437649 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437580 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-systemd-units\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.437649 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437598 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-host\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.437649 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437614 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-var-lib-cni-multus\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.437787 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437672 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-registration-dir\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.437787 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437684 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-systemd-units\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.437787 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437679 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-host\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.437787 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437742 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-device-dir\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.437787 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437782 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-sys-fs\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.437947 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437814 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-etc-openvswitch\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.437947 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437857 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-etc-openvswitch\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.437947 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437886 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-run-systemd\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.437947 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437906 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-systemd\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.437947 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437930 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-run-multus-certs\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.438104 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437952 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.438104 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437955 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-systemd\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.438104 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.437990 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql99h\" (UniqueName: \"kubernetes.io/projected/e3979fa5-6b05-4e90-abf0-ae3cb0286f0c-kube-api-access-ql99h\") pod \"node-resolver-s8cmz\" (UID: \"e3979fa5-6b05-4e90-abf0-ae3cb0286f0c\") " pod="openshift-dns/node-resolver-s8cmz" Jul 06 23:56:19.438104 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438019 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9d35a3e-274a-4e04-84f5-176b933aa822-iptables-alerter-script\") pod \"iptables-alerter-8t6pk\" (UID: \"c9d35a3e-274a-4e04-84f5-176b933aa822\") " pod="openshift-network-operator/iptables-alerter-8t6pk" Jul 06 23:56:19.438104 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438032 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-run-systemd\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.438104 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438046 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-run-netns\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.438104 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438072 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76bea87f-a057-4a59-8e4a-171542fa018a-kubelet-config\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438120 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76bea87f-a057-4a59-8e4a-171542fa018a-kubelet-config\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438136 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-run-netns\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438139 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-kubernetes\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438183 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38be6bda-4bd1-4d2f-912f-2c1f0b372ed0-agent-certs\") pod \"konnectivity-agent-tdb62\" (UID: \"38be6bda-4bd1-4d2f-912f-2c1f0b372ed0\") " pod="kube-system/konnectivity-agent-tdb62" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438188 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-kubernetes\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438212 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3f35205-31fd-4286-af00-264dd8d0adb3-multus-daemon-config\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438243 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-log-socket\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438266 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56059632-3851-4ea0-8a86-fcabfc509e65-host\") pod \"node-ca-4t5gn\" (UID: \"56059632-3851-4ea0-8a86-fcabfc509e65\") " pod="openshift-image-registry/node-ca-4t5gn" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438290 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438300 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-log-socket\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438316 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-cni-netd\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438342 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76bea87f-a057-4a59-8e4a-171542fa018a-dbus\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438365 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-modprobe-d\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438390 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-hostroot\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.438404 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438366 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-cni-netd\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438415 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90898b8a-6bbd-45ae-b92b-24033e583d13-env-overrides\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438457 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9d35a3e-274a-4e04-84f5-176b933aa822-host-slash\") pod \"iptables-alerter-8t6pk\" (UID: \"c9d35a3e-274a-4e04-84f5-176b933aa822\") " pod="openshift-network-operator/iptables-alerter-8t6pk" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438503 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-modprobe-d\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438488 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55bq\" (UniqueName: \"kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq\") pod \"network-check-target-bcn92\" (UID: \"388db162-a514-4da6-b4ad-398a94e22e01\") " pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438548 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9d35a3e-274a-4e04-84f5-176b933aa822-host-slash\") pod \"iptables-alerter-8t6pk\" (UID: \"c9d35a3e-274a-4e04-84f5-176b933aa822\") " pod="openshift-network-operator/iptables-alerter-8t6pk" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438550 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76bea87f-a057-4a59-8e4a-171542fa018a-dbus\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438564 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56059632-3851-4ea0-8a86-fcabfc509e65-serviceca\") pod \"node-ca-4t5gn\" (UID: \"56059632-3851-4ea0-8a86-fcabfc509e65\") " pod="openshift-image-registry/node-ca-4t5gn" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438596 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-socket-dir\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438630 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-lib-modules\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438647 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-cnibin\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438661 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-run-netns\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438663 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9d35a3e-274a-4e04-84f5-176b933aa822-iptables-alerter-script\") pod \"iptables-alerter-8t6pk\" (UID: \"c9d35a3e-274a-4e04-84f5-176b933aa822\") " pod="openshift-network-operator/iptables-alerter-8t6pk" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438688 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpgdk\" (UniqueName: \"kubernetes.io/projected/90898b8a-6bbd-45ae-b92b-24033e583d13-kube-api-access-jpgdk\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438785 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-lib-modules\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438820 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38be6bda-4bd1-4d2f-912f-2c1f0b372ed0-konnectivity-ca\") pod \"konnectivity-agent-tdb62\" (UID: \"38be6bda-4bd1-4d2f-912f-2c1f0b372ed0\") " pod="kube-system/konnectivity-agent-tdb62" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438856 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3f35205-31fd-4286-af00-264dd8d0adb3-cni-binary-copy\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.439308 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438882 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-multus-socket-dir-parent\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438910 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-cnibin\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438935 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.438964 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-run-openvswitch\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439007 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90898b8a-6bbd-45ae-b92b-24033e583d13-ovnkube-config\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439033 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90898b8a-6bbd-45ae-b92b-24033e583d13-ovn-node-metrics-cert\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439056 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-run\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439073 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-run-openvswitch\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439080 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-var-lib-kubelet\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439130 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-var-lib-kubelet\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439154 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-run\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439185 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90898b8a-6bbd-45ae-b92b-24033e583d13-env-overrides\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439204 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgt2p\" (UniqueName: \"kubernetes.io/projected/56059632-3851-4ea0-8a86-fcabfc509e65-kube-api-access-xgt2p\") pod \"node-ca-4t5gn\" (UID: \"56059632-3851-4ea0-8a86-fcabfc509e65\") " pod="openshift-image-registry/node-ca-4t5gn" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439252 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-etc-selinux\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-run-ovn\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439342 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-kubelet\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439349 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-run-ovn\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.440236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3979fa5-6b05-4e90-abf0-ae3cb0286f0c-tmp-dir\") pod \"node-resolver-s8cmz\" (UID: \"e3979fa5-6b05-4e90-abf0-ae3cb0286f0c\") " pod="openshift-dns/node-resolver-s8cmz" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439391 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-tuned\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439393 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-kubelet\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439418 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5pqc\" (UniqueName: \"kubernetes.io/projected/f3f35205-31fd-4286-af00-264dd8d0adb3-kube-api-access-h5pqc\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439463 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-slash\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439435 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439491 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-run-ovn-kubernetes\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439517 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6xdm\" (UniqueName: \"kubernetes.io/projected/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-kube-api-access-f6xdm\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439544 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn4nh\" (UniqueName: \"kubernetes.io/projected/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-kube-api-access-qn4nh\") pod \"network-metrics-daemon-mddqt\" (UID: \"b8d4e21f-d7f0-404d-a8e8-4da642d240c7\") " pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439558 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90898b8a-6bbd-45ae-b92b-24033e583d13-ovnkube-config\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439572 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr9ss\" (UniqueName: \"kubernetes.io/projected/c9d35a3e-274a-4e04-84f5-176b933aa822-kube-api-access-lr9ss\") pod \"iptables-alerter-8t6pk\" (UID: \"c9d35a3e-274a-4e04-84f5-176b933aa822\") " pod="openshift-network-operator/iptables-alerter-8t6pk" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439624 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90898b8a-6bbd-45ae-b92b-24033e583d13-ovnkube-script-lib\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439629 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-run-ovn-kubernetes\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439653 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr7nj\" (UniqueName: \"kubernetes.io/projected/f2662ed3-dc3a-402d-bce7-0657390f9350-kube-api-access-tr7nj\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439680 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-system-cni-dir\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439693 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-slash\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439704 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-var-lib-cni-bin\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.441080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439755 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-etc-kubernetes\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439762 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3979fa5-6b05-4e90-abf0-ae3cb0286f0c-tmp-dir\") pod \"node-resolver-s8cmz\" (UID: \"e3979fa5-6b05-4e90-abf0-ae3cb0286f0c\") " pod="openshift-dns/node-resolver-s8cmz" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439788 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqd4r\" (UniqueName: \"kubernetes.io/projected/bc142a49-37fc-405e-b7bf-5fb41e22f850-kube-api-access-xqd4r\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439815 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-var-lib-openvswitch\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439837 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-node-log\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439883 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-var-lib-kubelet\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439939 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-var-lib-openvswitch\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439945 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439890 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-node-log\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.439982 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440015 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3979fa5-6b05-4e90-abf0-ae3cb0286f0c-hosts-file\") pod \"node-resolver-s8cmz\" (UID: \"e3979fa5-6b05-4e90-abf0-ae3cb0286f0c\") " pod="openshift-dns/node-resolver-s8cmz" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440061 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3979fa5-6b05-4e90-abf0-ae3cb0286f0c-hosts-file\") pod \"node-resolver-s8cmz\" (UID: \"e3979fa5-6b05-4e90-abf0-ae3cb0286f0c\") " pod="openshift-dns/node-resolver-s8cmz" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440164 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-sysctl-d\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440192 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-multus-cni-dir\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440213 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-os-release\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440248 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-run-k8s-cni-cncf-io\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.442018 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-sysctl-conf\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440284 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-sysctl-d\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440291 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-sys\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440315 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90898b8a-6bbd-45ae-b92b-24033e583d13-ovnkube-script-lib\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440336 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-multus-conf-dir\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440344 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-sys\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440362 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-system-cni-dir\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440388 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-os-release\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440415 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-cni-bin\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440431 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-sysctl-conf\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440440 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440468 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-sysconfig\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440470 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90898b8a-6bbd-45ae-b92b-24033e583d13-host-cni-bin\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440551 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-sysconfig\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:19.440583 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.440579 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2662ed3-dc3a-402d-bce7-0657390f9350-tmp\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.442879 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:19.440660 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret podName:76bea87f-a057-4a59-8e4a-171542fa018a nodeName:}" failed. No retries permitted until 2026-07-06 23:56:19.940639551 +0000 UTC m=+3.081096607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret") pod "global-pull-secret-syncer-x5k59" (UID: "76bea87f-a057-4a59-8e4a-171542fa018a") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:19.443774 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.442906 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2662ed3-dc3a-402d-bce7-0657390f9350-tmp\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.443774 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.443051 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2662ed3-dc3a-402d-bce7-0657390f9350-etc-tuned\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.443774 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.443141 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90898b8a-6bbd-45ae-b92b-24033e583d13-ovn-node-metrics-cert\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.448747 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.448725 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql99h\" (UniqueName: \"kubernetes.io/projected/e3979fa5-6b05-4e90-abf0-ae3cb0286f0c-kube-api-access-ql99h\") pod \"node-resolver-s8cmz\" (UID: \"e3979fa5-6b05-4e90-abf0-ae3cb0286f0c\") " pod="openshift-dns/node-resolver-s8cmz" Jul 06 23:56:19.450408 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:19.450389 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:19.450524 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:19.450416 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:19.450524 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:19.450432 2568 projected.go:196] Error preparing data for projected volume kube-api-access-k55bq for pod openshift-network-diagnostics/network-check-target-bcn92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:19.450524 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:19.450488 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq podName:388db162-a514-4da6-b4ad-398a94e22e01 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:19.950470923 +0000 UTC m=+3.090927978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k55bq" (UniqueName: "kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq") pod "network-check-target-bcn92" (UID: "388db162-a514-4da6-b4ad-398a94e22e01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:19.452613 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.452592 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpgdk\" (UniqueName: \"kubernetes.io/projected/90898b8a-6bbd-45ae-b92b-24033e583d13-kube-api-access-jpgdk\") pod \"ovnkube-node-rg8hd\" (UID: \"90898b8a-6bbd-45ae-b92b-24033e583d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.452821 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.452805 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr7nj\" (UniqueName: \"kubernetes.io/projected/f2662ed3-dc3a-402d-bce7-0657390f9350-kube-api-access-tr7nj\") pod \"tuned-2vpk2\" (UID: \"f2662ed3-dc3a-402d-bce7-0657390f9350\") " pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.454542 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.454519 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr9ss\" (UniqueName: \"kubernetes.io/projected/c9d35a3e-274a-4e04-84f5-176b933aa822-kube-api-access-lr9ss\") pod \"iptables-alerter-8t6pk\" (UID: \"c9d35a3e-274a-4e04-84f5-176b933aa822\") " pod="openshift-network-operator/iptables-alerter-8t6pk" Jul 06 23:56:19.532957 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:19.532926 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1874cbf2171f4c31314a83eeadd2b7f1.slice/crio-700403ea0bdd5b68df7a8ba474a6924a3b6a5c61c265be2f5005776e6f13590b WatchSource:0}: Error finding container 700403ea0bdd5b68df7a8ba474a6924a3b6a5c61c265be2f5005776e6f13590b: Status 404 returned error can't find the container with id 700403ea0bdd5b68df7a8ba474a6924a3b6a5c61c265be2f5005776e6f13590b Jul 06 23:56:19.539456 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.539438 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 23:56:19.541467 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541444 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-etc-selinux\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.541559 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541490 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5pqc\" (UniqueName: \"kubernetes.io/projected/f3f35205-31fd-4286-af00-264dd8d0adb3-kube-api-access-h5pqc\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.541559 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541521 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6xdm\" (UniqueName: \"kubernetes.io/projected/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-kube-api-access-f6xdm\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.541559 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn4nh\" (UniqueName: \"kubernetes.io/projected/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-kube-api-access-qn4nh\") pod \"network-metrics-daemon-mddqt\" (UID: \"b8d4e21f-d7f0-404d-a8e8-4da642d240c7\") " pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:19.541726 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541581 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-system-cni-dir\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.541726 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541607 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-var-lib-cni-bin\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.541726 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541613 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-etc-selinux\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.541726 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541633 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-etc-kubernetes\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.541726 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541658 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqd4r\" (UniqueName: \"kubernetes.io/projected/bc142a49-37fc-405e-b7bf-5fb41e22f850-kube-api-access-xqd4r\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.541726 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541677 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-var-lib-cni-bin\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.541726 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541689 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-system-cni-dir\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.541726 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541696 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-var-lib-kubelet\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.541726 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541720 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-etc-kubernetes\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541741 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-var-lib-kubelet\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541747 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541786 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-multus-cni-dir\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541854 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-multus-cni-dir\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541870 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-os-release\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541910 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-run-k8s-cni-cncf-io\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541930 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541941 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-multus-conf-dir\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541968 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-system-cni-dir\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541979 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-os-release\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541993 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-multus-conf-dir\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541995 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-os-release\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.541970 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-run-k8s-cni-cncf-io\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542022 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-system-cni-dir\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542039 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542059 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs\") pod \"network-metrics-daemon-mddqt\" (UID: \"b8d4e21f-d7f0-404d-a8e8-4da642d240c7\") " pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:19.542110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542075 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-var-lib-cni-multus\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542102 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-registration-dir\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542121 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-device-dir\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542136 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-sys-fs\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542171 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-run-multus-certs\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542193 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542212 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38be6bda-4bd1-4d2f-912f-2c1f0b372ed0-agent-certs\") pod \"konnectivity-agent-tdb62\" (UID: \"38be6bda-4bd1-4d2f-912f-2c1f0b372ed0\") " pod="kube-system/konnectivity-agent-tdb62" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542224 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-registration-dir\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542232 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3f35205-31fd-4286-af00-264dd8d0adb3-multus-daemon-config\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:19.542140 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542242 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-device-dir\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542258 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56059632-3851-4ea0-8a86-fcabfc509e65-host\") pod \"node-ca-4t5gn\" (UID: \"56059632-3851-4ea0-8a86-fcabfc509e65\") " pod="openshift-image-registry/node-ca-4t5gn" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542266 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542283 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542286 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-var-lib-cni-multus\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:19.542300 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs podName:b8d4e21f-d7f0-404d-a8e8-4da642d240c7 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:20.042282271 +0000 UTC m=+3.182739319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs") pod "network-metrics-daemon-mddqt" (UID: "b8d4e21f-d7f0-404d-a8e8-4da642d240c7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542292 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-run-multus-certs\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.543014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542191 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-os-release\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542337 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-sys-fs\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542340 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56059632-3851-4ea0-8a86-fcabfc509e65-host\") pod \"node-ca-4t5gn\" (UID: \"56059632-3851-4ea0-8a86-fcabfc509e65\") " pod="openshift-image-registry/node-ca-4t5gn" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542376 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-hostroot\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542448 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56059632-3851-4ea0-8a86-fcabfc509e65-serviceca\") pod \"node-ca-4t5gn\" (UID: \"56059632-3851-4ea0-8a86-fcabfc509e65\") " pod="openshift-image-registry/node-ca-4t5gn" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542478 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-socket-dir\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542581 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-cnibin\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542823 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542827 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542885 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-run-netns\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542896 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-hostroot\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542909 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-cnibin\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542921 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3f35205-31fd-4286-af00-264dd8d0adb3-multus-daemon-config\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542935 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38be6bda-4bd1-4d2f-912f-2c1f0b372ed0-konnectivity-ca\") pod \"konnectivity-agent-tdb62\" (UID: \"38be6bda-4bd1-4d2f-912f-2c1f0b372ed0\") " pod="kube-system/konnectivity-agent-tdb62" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.542959 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-host-run-netns\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.543015 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56059632-3851-4ea0-8a86-fcabfc509e65-serviceca\") pod \"node-ca-4t5gn\" (UID: \"56059632-3851-4ea0-8a86-fcabfc509e65\") " pod="openshift-image-registry/node-ca-4t5gn" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.543138 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc142a49-37fc-405e-b7bf-5fb41e22f850-socket-dir\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.543148 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3f35205-31fd-4286-af00-264dd8d0adb3-cni-binary-copy\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.543841 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.543183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-multus-socket-dir-parent\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.544543 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.543209 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-cnibin\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.544543 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.543234 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.544543 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.543265 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgt2p\" (UniqueName: \"kubernetes.io/projected/56059632-3851-4ea0-8a86-fcabfc509e65-kube-api-access-xgt2p\") pod \"node-ca-4t5gn\" (UID: \"56059632-3851-4ea0-8a86-fcabfc509e65\") " pod="openshift-image-registry/node-ca-4t5gn" Jul 06 23:56:19.544543 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.543273 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-cnibin\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.544543 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.543330 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3f35205-31fd-4286-af00-264dd8d0adb3-multus-socket-dir-parent\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.544543 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.543512 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38be6bda-4bd1-4d2f-912f-2c1f0b372ed0-konnectivity-ca\") pod \"konnectivity-agent-tdb62\" (UID: \"38be6bda-4bd1-4d2f-912f-2c1f0b372ed0\") " pod="kube-system/konnectivity-agent-tdb62" Jul 06 23:56:19.544543 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.543753 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3f35205-31fd-4286-af00-264dd8d0adb3-cni-binary-copy\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.544543 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.543761 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.545105 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:19.545085 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa7c948617b9fac2532d316a93a8386.slice/crio-ae967415a17281eb2ce887a0e2c9ae50ac64431f823bccd5c6325be5d4bd0dbe WatchSource:0}: Error finding container ae967415a17281eb2ce887a0e2c9ae50ac64431f823bccd5c6325be5d4bd0dbe: Status 404 returned error can't find the container with id ae967415a17281eb2ce887a0e2c9ae50ac64431f823bccd5c6325be5d4bd0dbe Jul 06 23:56:19.545305 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.545286 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38be6bda-4bd1-4d2f-912f-2c1f0b372ed0-agent-certs\") pod \"konnectivity-agent-tdb62\" (UID: \"38be6bda-4bd1-4d2f-912f-2c1f0b372ed0\") " pod="kube-system/konnectivity-agent-tdb62" Jul 06 23:56:19.550635 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.550616 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5pqc\" (UniqueName: \"kubernetes.io/projected/f3f35205-31fd-4286-af00-264dd8d0adb3-kube-api-access-h5pqc\") pod \"multus-fnfp4\" (UID: \"f3f35205-31fd-4286-af00-264dd8d0adb3\") " pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.551904 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.551883 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn4nh\" (UniqueName: \"kubernetes.io/projected/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-kube-api-access-qn4nh\") pod \"network-metrics-daemon-mddqt\" (UID: \"b8d4e21f-d7f0-404d-a8e8-4da642d240c7\") " pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:19.552213 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.552187 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgt2p\" (UniqueName: \"kubernetes.io/projected/56059632-3851-4ea0-8a86-fcabfc509e65-kube-api-access-xgt2p\") pod \"node-ca-4t5gn\" (UID: \"56059632-3851-4ea0-8a86-fcabfc509e65\") " pod="openshift-image-registry/node-ca-4t5gn" Jul 06 23:56:19.552740 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.552718 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqd4r\" (UniqueName: \"kubernetes.io/projected/bc142a49-37fc-405e-b7bf-5fb41e22f850-kube-api-access-xqd4r\") pod \"aws-ebs-csi-driver-node-sdb7w\" (UID: \"bc142a49-37fc-405e-b7bf-5fb41e22f850\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.553093 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.553078 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6xdm\" (UniqueName: \"kubernetes.io/projected/5ca7a76e-7c60-4e0c-bbda-a1edb3def89d-kube-api-access-f6xdm\") pod \"multus-additional-cni-plugins-qlz2q\" (UID: \"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d\") " pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.637975 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.637924 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:19.642192 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.642180 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s8cmz" Jul 06 23:56:19.651824 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:19.651800 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3979fa5_6b05_4e90_abf0_ae3cb0286f0c.slice/crio-dde1c1aa74dce5bc30aabf92aca4740f179779e38f093f6ac04dffbc305310e3 WatchSource:0}: Error finding container dde1c1aa74dce5bc30aabf92aca4740f179779e38f093f6ac04dffbc305310e3: Status 404 returned error can't find the container with id dde1c1aa74dce5bc30aabf92aca4740f179779e38f093f6ac04dffbc305310e3 Jul 06 23:56:19.663124 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.663107 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8t6pk" Jul 06 23:56:19.671182 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:19.671165 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d35a3e_274a_4e04_84f5_176b933aa822.slice/crio-cf3e0eb945f7077ba77b00af9d8e5e68d88200bcd7a5b73c56975a60a4dff49e WatchSource:0}: Error finding container cf3e0eb945f7077ba77b00af9d8e5e68d88200bcd7a5b73c56975a60a4dff49e: Status 404 returned error can't find the container with id cf3e0eb945f7077ba77b00af9d8e5e68d88200bcd7a5b73c56975a60a4dff49e Jul 06 23:56:19.684693 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.684677 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:19.692974 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:19.692956 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90898b8a_6bbd_45ae_b92b_24033e583d13.slice/crio-6d1c2525c2ba1238f424b54c87c090f433680769a72494b8fdadbd95087cb60b WatchSource:0}: Error finding container 6d1c2525c2ba1238f424b54c87c090f433680769a72494b8fdadbd95087cb60b: Status 404 returned error can't find the container with id 6d1c2525c2ba1238f424b54c87c090f433680769a72494b8fdadbd95087cb60b Jul 06 23:56:19.694530 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.694510 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" Jul 06 23:56:19.699555 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.699538 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fnfp4" Jul 06 23:56:19.703007 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:19.702991 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2662ed3_dc3a_402d_bce7_0657390f9350.slice/crio-4be73437c75e8e4bd0000201aa220c12b7101c804e520d57a9846dbfe6a59f11 WatchSource:0}: Error finding container 4be73437c75e8e4bd0000201aa220c12b7101c804e520d57a9846dbfe6a59f11: Status 404 returned error can't find the container with id 4be73437c75e8e4bd0000201aa220c12b7101c804e520d57a9846dbfe6a59f11 Jul 06 23:56:19.708305 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:19.708286 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f35205_31fd_4286_af00_264dd8d0adb3.slice/crio-91d9978a2065c58406fa0262959b0517a9056149c6da2c742c65f9067a54ed8a WatchSource:0}: Error finding container 91d9978a2065c58406fa0262959b0517a9056149c6da2c742c65f9067a54ed8a: Status 404 returned error can't find the container with id 91d9978a2065c58406fa0262959b0517a9056149c6da2c742c65f9067a54ed8a Jul 06 23:56:19.728543 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.728523 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" Jul 06 23:56:19.729809 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.729793 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:19.737195 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:19.737162 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc142a49_37fc_405e_b7bf_5fb41e22f850.slice/crio-11f26467fbabfc5f8c7876f667243f7ce7c4d8afe280dadae0e8eb8f64d43a71 WatchSource:0}: Error finding container 11f26467fbabfc5f8c7876f667243f7ce7c4d8afe280dadae0e8eb8f64d43a71: Status 404 returned error can't find the container with id 11f26467fbabfc5f8c7876f667243f7ce7c4d8afe280dadae0e8eb8f64d43a71 Jul 06 23:56:19.743264 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.743245 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qlz2q" Jul 06 23:56:19.752600 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:19.752580 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca7a76e_7c60_4e0c_bbda_a1edb3def89d.slice/crio-063fade0959806eb3d65ca6eb629890c364c8dbe2c4280ff15c309928360caa2 WatchSource:0}: Error finding container 063fade0959806eb3d65ca6eb629890c364c8dbe2c4280ff15c309928360caa2: Status 404 returned error can't find the container with id 063fade0959806eb3d65ca6eb629890c364c8dbe2c4280ff15c309928360caa2 Jul 06 23:56:19.765123 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.765106 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4t5gn" Jul 06 23:56:19.770175 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.770157 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tdb62" Jul 06 23:56:19.774678 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:19.774659 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56059632_3851_4ea0_8a86_fcabfc509e65.slice/crio-c06e5e13428fe6dc40450f77d5bcc041b0fb2572d3c55b43d7ca5b1f6517ea60 WatchSource:0}: Error finding container c06e5e13428fe6dc40450f77d5bcc041b0fb2572d3c55b43d7ca5b1f6517ea60: Status 404 returned error can't find the container with id c06e5e13428fe6dc40450f77d5bcc041b0fb2572d3c55b43d7ca5b1f6517ea60 Jul 06 23:56:19.779238 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:19.779217 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38be6bda_4bd1_4d2f_912f_2c1f0b372ed0.slice/crio-09b85aa399ee904c8b91ce2e173423fd202d14c7dea151bdb3d8ef2e6e4e52d4 WatchSource:0}: Error finding container 09b85aa399ee904c8b91ce2e173423fd202d14c7dea151bdb3d8ef2e6e4e52d4: Status 404 returned error can't find the container with id 09b85aa399ee904c8b91ce2e173423fd202d14c7dea151bdb3d8ef2e6e4e52d4 Jul 06 23:56:19.945575 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:19.945519 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:19.945686 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:19.945630 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:19.945754 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:19.945695 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret podName:76bea87f-a057-4a59-8e4a-171542fa018a nodeName:}" failed. No retries permitted until 2026-07-06 23:56:20.945674286 +0000 UTC m=+4.086131339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret") pod "global-pull-secret-syncer-x5k59" (UID: "76bea87f-a057-4a59-8e4a-171542fa018a") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:20.046055 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.045941 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55bq\" (UniqueName: \"kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq\") pod \"network-check-target-bcn92\" (UID: \"388db162-a514-4da6-b4ad-398a94e22e01\") " pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:20.046055 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.046026 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs\") pod \"network-metrics-daemon-mddqt\" (UID: \"b8d4e21f-d7f0-404d-a8e8-4da642d240c7\") " pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:20.046271 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:20.046064 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:20.046271 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:20.046084 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:20.046271 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:20.046100 2568 projected.go:196] Error preparing data for projected volume kube-api-access-k55bq for pod openshift-network-diagnostics/network-check-target-bcn92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:20.046271 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:20.046116 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:20.046271 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:20.046160 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs podName:b8d4e21f-d7f0-404d-a8e8-4da642d240c7 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:21.04614241 +0000 UTC m=+4.186599450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs") pod "network-metrics-daemon-mddqt" (UID: "b8d4e21f-d7f0-404d-a8e8-4da642d240c7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:20.046271 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:20.046178 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq podName:388db162-a514-4da6-b4ad-398a94e22e01 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:21.046167975 +0000 UTC m=+4.186625011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k55bq" (UniqueName: "kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq") pod "network-check-target-bcn92" (UID: "388db162-a514-4da6-b4ad-398a94e22e01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:20.372839 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.372723 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 23:51:19 +0000 UTC" deadline="2028-01-23 09:54:04.719162554 +0000 UTC" Jul 06 23:56:20.372839 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.372796 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13569h57m44.346370369s" Jul 06 23:56:20.480854 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.480816 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:20.481048 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:20.480971 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:20.498241 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.498159 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" event={"ID":"90898b8a-6bbd-45ae-b92b-24033e583d13","Type":"ContainerStarted","Data":"6d1c2525c2ba1238f424b54c87c090f433680769a72494b8fdadbd95087cb60b"} Jul 06 23:56:20.514347 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.514309 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8t6pk" event={"ID":"c9d35a3e-274a-4e04-84f5-176b933aa822","Type":"ContainerStarted","Data":"cf3e0eb945f7077ba77b00af9d8e5e68d88200bcd7a5b73c56975a60a4dff49e"} Jul 06 23:56:20.520415 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.518924 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fnfp4" event={"ID":"f3f35205-31fd-4286-af00-264dd8d0adb3","Type":"ContainerStarted","Data":"91d9978a2065c58406fa0262959b0517a9056149c6da2c742c65f9067a54ed8a"} Jul 06 23:56:20.545022 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.544914 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s8cmz" event={"ID":"e3979fa5-6b05-4e90-abf0-ae3cb0286f0c","Type":"ContainerStarted","Data":"dde1c1aa74dce5bc30aabf92aca4740f179779e38f093f6ac04dffbc305310e3"} Jul 06 23:56:20.573539 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.568736 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" event={"ID":"2fa7c948617b9fac2532d316a93a8386","Type":"ContainerStarted","Data":"ae967415a17281eb2ce887a0e2c9ae50ac64431f823bccd5c6325be5d4bd0dbe"} Jul 06 23:56:20.589215 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.589142 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" event={"ID":"1874cbf2171f4c31314a83eeadd2b7f1","Type":"ContainerStarted","Data":"700403ea0bdd5b68df7a8ba474a6924a3b6a5c61c265be2f5005776e6f13590b"} Jul 06 23:56:20.623317 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.623231 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tdb62" event={"ID":"38be6bda-4bd1-4d2f-912f-2c1f0b372ed0","Type":"ContainerStarted","Data":"09b85aa399ee904c8b91ce2e173423fd202d14c7dea151bdb3d8ef2e6e4e52d4"} Jul 06 23:56:20.625385 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.625358 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4t5gn" event={"ID":"56059632-3851-4ea0-8a86-fcabfc509e65","Type":"ContainerStarted","Data":"c06e5e13428fe6dc40450f77d5bcc041b0fb2572d3c55b43d7ca5b1f6517ea60"} Jul 06 23:56:20.645847 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.645810 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlz2q" event={"ID":"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d","Type":"ContainerStarted","Data":"063fade0959806eb3d65ca6eb629890c364c8dbe2c4280ff15c309928360caa2"} Jul 06 23:56:20.662313 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.662226 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" event={"ID":"bc142a49-37fc-405e-b7bf-5fb41e22f850","Type":"ContainerStarted","Data":"11f26467fbabfc5f8c7876f667243f7ce7c4d8afe280dadae0e8eb8f64d43a71"} Jul 06 23:56:20.674825 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.674790 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" event={"ID":"f2662ed3-dc3a-402d-bce7-0657390f9350","Type":"ContainerStarted","Data":"4be73437c75e8e4bd0000201aa220c12b7101c804e520d57a9846dbfe6a59f11"} Jul 06 23:56:20.954408 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:20.953812 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:20.954408 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:20.953992 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:20.954408 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:20.954052 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret podName:76bea87f-a057-4a59-8e4a-171542fa018a nodeName:}" failed. No retries permitted until 2026-07-06 23:56:22.954033658 +0000 UTC m=+6.094490698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret") pod "global-pull-secret-syncer-x5k59" (UID: "76bea87f-a057-4a59-8e4a-171542fa018a") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:21.054549 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:21.054511 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs\") pod \"network-metrics-daemon-mddqt\" (UID: \"b8d4e21f-d7f0-404d-a8e8-4da642d240c7\") " pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:21.054742 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:21.054575 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55bq\" (UniqueName: \"kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq\") pod \"network-check-target-bcn92\" (UID: \"388db162-a514-4da6-b4ad-398a94e22e01\") " pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:21.054742 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:21.054720 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:21.054742 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:21.054742 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:21.054917 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:21.054755 2568 projected.go:196] Error preparing data for projected volume kube-api-access-k55bq for pod openshift-network-diagnostics/network-check-target-bcn92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:21.054917 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:21.054809 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq podName:388db162-a514-4da6-b4ad-398a94e22e01 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:23.054790192 +0000 UTC m=+6.195247234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k55bq" (UniqueName: "kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq") pod "network-check-target-bcn92" (UID: "388db162-a514-4da6-b4ad-398a94e22e01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:21.055277 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:21.055198 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:21.055277 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:21.055249 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs podName:b8d4e21f-d7f0-404d-a8e8-4da642d240c7 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:23.055234338 +0000 UTC m=+6.195691377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs") pod "network-metrics-daemon-mddqt" (UID: "b8d4e21f-d7f0-404d-a8e8-4da642d240c7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:21.373049 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:21.372927 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 23:51:19 +0000 UTC" deadline="2028-03-21 19:02:30.746325992 +0000 UTC" Jul 06 23:56:21.373049 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:21.372997 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14971h6m9.373332839s" Jul 06 23:56:21.481719 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:21.481683 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:21.481883 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:21.481723 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:21.481883 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:21.481857 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:21.482392 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:21.482363 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:22.481371 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:22.481328 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:22.481888 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:22.481506 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:22.971545 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:22.971504 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:22.971761 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:22.971679 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:22.971761 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:22.971747 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret podName:76bea87f-a057-4a59-8e4a-171542fa018a nodeName:}" failed. No retries permitted until 2026-07-06 23:56:26.97172683 +0000 UTC m=+10.112183893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret") pod "global-pull-secret-syncer-x5k59" (UID: "76bea87f-a057-4a59-8e4a-171542fa018a") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:23.072968 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:23.072024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs\") pod \"network-metrics-daemon-mddqt\" (UID: \"b8d4e21f-d7f0-404d-a8e8-4da642d240c7\") " pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:23.072968 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:23.072094 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55bq\" (UniqueName: \"kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq\") pod \"network-check-target-bcn92\" (UID: \"388db162-a514-4da6-b4ad-398a94e22e01\") " pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:23.072968 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:23.072289 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:23.072968 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:23.072308 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:23.072968 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:23.072323 2568 projected.go:196] Error preparing data for projected volume kube-api-access-k55bq for pod openshift-network-diagnostics/network-check-target-bcn92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:23.072968 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:23.072384 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq podName:388db162-a514-4da6-b4ad-398a94e22e01 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:27.07236419 +0000 UTC m=+10.212821226 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k55bq" (UniqueName: "kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq") pod "network-check-target-bcn92" (UID: "388db162-a514-4da6-b4ad-398a94e22e01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:23.072968 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:23.072833 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:23.072968 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:23.072888 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs podName:b8d4e21f-d7f0-404d-a8e8-4da642d240c7 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:27.072871679 +0000 UTC m=+10.213328719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs") pod "network-metrics-daemon-mddqt" (UID: "b8d4e21f-d7f0-404d-a8e8-4da642d240c7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:23.481633 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:23.481597 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:23.482152 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:23.481735 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:23.483779 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:23.483580 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:23.483779 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:23.483736 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:24.480969 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:24.480421 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:24.480969 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:24.480574 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:25.481941 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:25.481424 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:25.481941 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:25.481583 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:25.482536 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:25.482007 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:25.482536 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:25.482117 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:26.480347 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:26.480257 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:26.480590 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:26.480414 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:27.006035 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:27.005992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:27.006539 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:27.006201 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:27.006539 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:27.006265 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret podName:76bea87f-a057-4a59-8e4a-171542fa018a nodeName:}" failed. No retries permitted until 2026-07-06 23:56:35.006245188 +0000 UTC m=+18.146702306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret") pod "global-pull-secret-syncer-x5k59" (UID: "76bea87f-a057-4a59-8e4a-171542fa018a") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:27.106650 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:27.106608 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55bq\" (UniqueName: \"kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq\") pod \"network-check-target-bcn92\" (UID: \"388db162-a514-4da6-b4ad-398a94e22e01\") " pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:27.106837 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:27.106814 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs\") pod \"network-metrics-daemon-mddqt\" (UID: \"b8d4e21f-d7f0-404d-a8e8-4da642d240c7\") " pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:27.106904 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:27.106851 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:27.106904 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:27.106873 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:27.106904 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:27.106888 2568 projected.go:196] Error preparing data for projected volume kube-api-access-k55bq for pod openshift-network-diagnostics/network-check-target-bcn92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:27.107044 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:27.106956 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq podName:388db162-a514-4da6-b4ad-398a94e22e01 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:35.106936537 +0000 UTC m=+18.247393576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k55bq" (UniqueName: "kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq") pod "network-check-target-bcn92" (UID: "388db162-a514-4da6-b4ad-398a94e22e01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:27.107044 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:27.106963 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:27.107044 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:27.107024 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs podName:b8d4e21f-d7f0-404d-a8e8-4da642d240c7 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:35.107010942 +0000 UTC m=+18.247467977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs") pod "network-metrics-daemon-mddqt" (UID: "b8d4e21f-d7f0-404d-a8e8-4da642d240c7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:27.481836 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:27.481365 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:27.481836 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:27.481499 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:27.481836 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:27.481558 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:27.481836 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:27.481643 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:28.480438 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:28.480398 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:28.480887 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:28.480536 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:29.481188 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:29.481153 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:29.481584 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:29.481290 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:29.481584 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:29.481153 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:29.481584 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:29.481393 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:30.480324 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:30.480295 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:30.480514 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:30.480405 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:31.481304 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:31.481045 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:31.481797 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:31.481096 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:31.481797 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:31.481432 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:31.481797 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:31.481492 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:32.481063 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:32.481031 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:32.481217 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:32.481147 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:33.483604 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:33.483578 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:33.483604 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:33.483594 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:33.484083 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:33.483680 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:33.484083 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:33.483814 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:34.480283 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:34.480245 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:34.480497 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:34.480397 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:35.068849 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:35.068811 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:35.069313 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:35.068972 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:35.069313 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:35.069048 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret podName:76bea87f-a057-4a59-8e4a-171542fa018a nodeName:}" failed. No retries permitted until 2026-07-06 23:56:51.069029103 +0000 UTC m=+34.209486157 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret") pod "global-pull-secret-syncer-x5k59" (UID: "76bea87f-a057-4a59-8e4a-171542fa018a") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:35.170142 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:35.170105 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55bq\" (UniqueName: \"kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq\") pod \"network-check-target-bcn92\" (UID: \"388db162-a514-4da6-b4ad-398a94e22e01\") " pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:35.170337 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:35.170190 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs\") pod \"network-metrics-daemon-mddqt\" (UID: \"b8d4e21f-d7f0-404d-a8e8-4da642d240c7\") " pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:35.170337 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:35.170305 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:35.170457 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:35.170366 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs podName:b8d4e21f-d7f0-404d-a8e8-4da642d240c7 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:51.17034497 +0000 UTC m=+34.310802026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs") pod "network-metrics-daemon-mddqt" (UID: "b8d4e21f-d7f0-404d-a8e8-4da642d240c7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:35.170457 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:35.170303 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:35.170457 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:35.170433 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:35.170457 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:35.170448 2568 projected.go:196] Error preparing data for projected volume kube-api-access-k55bq for pod openshift-network-diagnostics/network-check-target-bcn92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:35.170642 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:35.170502 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq podName:388db162-a514-4da6-b4ad-398a94e22e01 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:51.170487041 +0000 UTC m=+34.310944076 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k55bq" (UniqueName: "kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq") pod "network-check-target-bcn92" (UID: "388db162-a514-4da6-b4ad-398a94e22e01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:35.480413 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:35.480377 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:35.480729 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:35.480380 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:35.480729 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:35.480501 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:35.480729 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:35.480571 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:36.480610 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:36.480574 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:36.481006 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:36.480716 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:37.481053 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:37.481019 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:37.481545 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:37.481123 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:37.481545 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:37.481179 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:37.481545 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:37.481261 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:38.481149 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:38.481109 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:38.481574 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:38.481264 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:39.481319 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:39.481284 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:39.481789 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:39.481433 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:39.481789 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:39.481475 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:39.481789 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:39.481579 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:40.480397 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:40.480363 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:40.480666 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:40.480497 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:40.733892 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:40.733211 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" event={"ID":"f2662ed3-dc3a-402d-bce7-0657390f9350","Type":"ContainerStarted","Data":"dcba22d7edcb2244fa2d7cbe7830cc01c61876e94b297d9852dedf97c262854b"} Jul 06 23:56:40.737275 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:40.737223 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" event={"ID":"90898b8a-6bbd-45ae-b92b-24033e583d13","Type":"ContainerStarted","Data":"dd5539f577c38643851b4936837303395575f228d8f1e693c3648add948e22c9"} Jul 06 23:56:40.743598 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:40.743560 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fnfp4" event={"ID":"f3f35205-31fd-4286-af00-264dd8d0adb3","Type":"ContainerStarted","Data":"3071b42aaf32c9ac3720f8672aa846a06297e30d354d5c7cce15066d3093114e"} Jul 06 23:56:40.754445 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:40.754305 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" event={"ID":"2fa7c948617b9fac2532d316a93a8386","Type":"ContainerStarted","Data":"536385625ccf12964f53b24cf3caa6d0f1a9a859845fd6630da00189817c8f20"} Jul 06 23:56:40.761838 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:40.761640 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2vpk2" podStartSLOduration=1.935023675 podStartE2EDuration="22.761622658s" totalImagesPullingTime="20.826598983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:40.760569771 +0000 UTC m=+23.901026827" watchObservedRunningTime="2026-07-06 23:56:40.761622658 +0000 UTC m=+23.902079716" Jul 06 23:56:40.783287 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:40.783243 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fnfp4" podStartSLOduration=1.908810857 podStartE2EDuration="22.783228749s" totalImagesPullingTime="20.874417892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:40.782267956 +0000 UTC m=+23.922725011" watchObservedRunningTime="2026-07-06 23:56:40.783228749 +0000 UTC m=+23.923685805" Jul 06 23:56:40.800188 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:40.800135 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" podStartSLOduration=21.800115037 podStartE2EDuration="21.800115037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:40.79966172 +0000 UTC m=+23.940118776" watchObservedRunningTime="2026-07-06 23:56:40.800115037 +0000 UTC m=+23.940572094" Jul 06 23:56:41.481214 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.481185 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:41.481392 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:41.481287 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:41.481392 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.481317 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:41.481392 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:41.481380 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:41.758327 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.758132 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" event={"ID":"bc142a49-37fc-405e-b7bf-5fb41e22f850","Type":"ContainerStarted","Data":"bfd163427bbd717a07f5f1c00fbbc2f682c5005fb84c5622412e3bbf3fdd6112"} Jul 06 23:56:41.761239 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.761212 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" event={"ID":"90898b8a-6bbd-45ae-b92b-24033e583d13","Type":"ContainerStarted","Data":"be7f356a9130a60d365babc6f5e2c4fe6067498a9c3306338df366174ef2b7e0"} Jul 06 23:56:41.761366 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.761298 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" event={"ID":"90898b8a-6bbd-45ae-b92b-24033e583d13","Type":"ContainerStarted","Data":"c2be0438785d5cc9529aa4bd26509aefe74fc58ad247503f822c461206f17c0a"} Jul 06 23:56:41.761366 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.761359 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" event={"ID":"90898b8a-6bbd-45ae-b92b-24033e583d13","Type":"ContainerStarted","Data":"1a7ccb36635abd92860694f3b6bc9a31366a343eb07c90b98fca64cb01d6851d"} Jul 06 23:56:41.761516 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.761397 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" event={"ID":"90898b8a-6bbd-45ae-b92b-24033e583d13","Type":"ContainerStarted","Data":"076108da61182f80c44bb51dc486b95eee3b1fb11b0bac88458694012f180ab1"} Jul 06 23:56:41.761516 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.761470 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" event={"ID":"90898b8a-6bbd-45ae-b92b-24033e583d13","Type":"ContainerStarted","Data":"58f6b95e80b891442b90f581b779548023f28e73f520651dadeaa207d8639b0a"} Jul 06 23:56:41.762791 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.762765 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8t6pk" event={"ID":"c9d35a3e-274a-4e04-84f5-176b933aa822","Type":"ContainerStarted","Data":"8be8a715e36f40bc5a0fc71940eee5042ee6749ccd60586ee1e1ff5a21b89afb"} Jul 06 23:56:41.764434 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.764396 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s8cmz" event={"ID":"e3979fa5-6b05-4e90-abf0-ae3cb0286f0c","Type":"ContainerStarted","Data":"ce0bbd62dbf064e287420aad7ec24af0f81d52892a3396a332afacba85dd71af"} Jul 06 23:56:41.766044 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.766020 2568 generic.go:356] "Generic (PLEG): container finished" podID="1874cbf2171f4c31314a83eeadd2b7f1" containerID="d12c331dde8058f7676a7254dbd86d4cdaf917064fab6557aa5d362996dfecaf" exitCode=0 Jul 06 23:56:41.766163 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.766074 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" event={"ID":"1874cbf2171f4c31314a83eeadd2b7f1","Type":"ContainerDied","Data":"d12c331dde8058f7676a7254dbd86d4cdaf917064fab6557aa5d362996dfecaf"} Jul 06 23:56:41.767622 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.767597 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tdb62" event={"ID":"38be6bda-4bd1-4d2f-912f-2c1f0b372ed0","Type":"ContainerStarted","Data":"842b256aa22e7a0d1d4a277918cf83968471feafe381af0a0a62bd26981cdc11"} Jul 06 23:56:41.769047 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.769028 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4t5gn" event={"ID":"56059632-3851-4ea0-8a86-fcabfc509e65","Type":"ContainerStarted","Data":"f862092cf4b8dd89c10c9cfbcb3db68448cf18e8382fb733c7863dee2cfe85ac"} Jul 06 23:56:41.770213 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.770194 2568 generic.go:356] "Generic (PLEG): container finished" podID="5ca7a76e-7c60-4e0c-bbda-a1edb3def89d" containerID="52304292b09861bb817bb33d6af0f5bf4762c4e2ba038c5de93a3e71cbf3b700" exitCode=0 Jul 06 23:56:41.770307 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.770253 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlz2q" event={"ID":"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d","Type":"ContainerDied","Data":"52304292b09861bb817bb33d6af0f5bf4762c4e2ba038c5de93a3e71cbf3b700"} Jul 06 23:56:41.788256 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.788222 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8t6pk" podStartSLOduration=2.944252125 podStartE2EDuration="23.78821058s" totalImagesPullingTime="20.843958455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:41.787866085 +0000 UTC m=+24.928323141" watchObservedRunningTime="2026-07-06 23:56:41.78821058 +0000 UTC m=+24.928667635" Jul 06 23:56:41.843713 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.843668 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4t5gn" podStartSLOduration=3.102616699 podStartE2EDuration="23.843654416s" totalImagesPullingTime="20.741037717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:41.827330664 +0000 UTC m=+24.967787722" watchObservedRunningTime="2026-07-06 23:56:41.843654416 +0000 UTC m=+24.984111472" Jul 06 23:56:41.844215 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.844180 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tdb62" podStartSLOduration=11.760888807 podStartE2EDuration="23.844172554s" totalImagesPullingTime="12.083283747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:41.843775067 +0000 UTC m=+24.984232104" watchObservedRunningTime="2026-07-06 23:56:41.844172554 +0000 UTC m=+24.984629609" Jul 06 23:56:41.877178 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:41.877138 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s8cmz" podStartSLOduration=3.013293537 podStartE2EDuration="23.877120238s" totalImagesPullingTime="20.863826701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:41.87654394 +0000 UTC m=+25.017000991" watchObservedRunningTime="2026-07-06 23:56:41.877120238 +0000 UTC m=+25.017577295" Jul 06 23:56:42.442642 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:42.442610 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 23:56:42.480389 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:42.480361 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:42.480559 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:42.480485 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:42.774961 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:42.774924 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" event={"ID":"bc142a49-37fc-405e-b7bf-5fb41e22f850","Type":"ContainerStarted","Data":"fdb90b91228596b8e106fd272baca7a087929b08e6bb2ce25bc0755a2782f61f"} Jul 06 23:56:42.777101 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:42.777068 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" event={"ID":"1874cbf2171f4c31314a83eeadd2b7f1","Type":"ContainerStarted","Data":"a6992ab0a265d497238afc8fe8449753e3a5823ae8cee65f0dceef306425b3a0"} Jul 06 23:56:42.793832 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:42.793801 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tdb62" Jul 06 23:56:42.794646 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:42.794615 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tdb62" Jul 06 23:56:42.796161 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:42.796122 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" podStartSLOduration=23.796109553 podStartE2EDuration="23.796109553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:42.795575938 +0000 UTC m=+25.936032993" watchObservedRunningTime="2026-07-06 23:56:42.796109553 +0000 UTC m=+25.936566609" Jul 06 23:56:43.410853 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:43.410769 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T23:56:42.442648274Z","UUID":"e472c3f8-1520-4a1b-93d2-b7292d3457d7","Handler":null,"Name":"","Endpoint":""} Jul 06 23:56:43.413323 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:43.413298 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 23:56:43.413458 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:43.413346 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 23:56:43.480869 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:43.480837 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:43.481030 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:43.480897 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:43.481091 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:43.480990 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:43.481150 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:43.481089 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:43.782415 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:43.782373 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" event={"ID":"bc142a49-37fc-405e-b7bf-5fb41e22f850","Type":"ContainerStarted","Data":"9c539c86858ef5ad80870901cf43fe443e5c6e29cc454184448824cdf2a976c5"} Jul 06 23:56:43.786271 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:43.786242 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" event={"ID":"90898b8a-6bbd-45ae-b92b-24033e583d13","Type":"ContainerStarted","Data":"3848140aaed7e4d8339ac60827d81bdd41e59f44205b6bd87c15dcf33283712b"} Jul 06 23:56:43.786437 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:43.786373 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tdb62" Jul 06 23:56:43.786942 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:43.786923 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tdb62" Jul 06 23:56:43.802566 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:43.802523 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sdb7w" podStartSLOduration=2.734826679 podStartE2EDuration="25.802506241s" totalImagesPullingTime="23.067679562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:43.80232097 +0000 UTC m=+26.942778039" watchObservedRunningTime="2026-07-06 23:56:43.802506241 +0000 UTC m=+26.942963297" Jul 06 23:56:44.480700 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:44.480649 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:44.480894 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:44.480808 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:45.481171 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:45.481133 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:45.481659 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:45.481272 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:45.481659 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:45.481361 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:45.481659 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:45.481489 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:46.480806 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:46.480617 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:46.481032 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:46.480885 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:46.794575 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:46.794492 2568 generic.go:356] "Generic (PLEG): container finished" podID="5ca7a76e-7c60-4e0c-bbda-a1edb3def89d" containerID="a837263565000067406a4d68206e66a0be6ffb5d3a4ec476fcffc5b891785f0e" exitCode=0 Jul 06 23:56:46.795296 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:46.794585 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlz2q" event={"ID":"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d","Type":"ContainerDied","Data":"a837263565000067406a4d68206e66a0be6ffb5d3a4ec476fcffc5b891785f0e"} Jul 06 23:56:46.798174 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:46.798150 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" event={"ID":"90898b8a-6bbd-45ae-b92b-24033e583d13","Type":"ContainerStarted","Data":"cc82a31ee0c2b576ca8104f78a7697641082dc502a30ba38519a2a3243bc772a"} Jul 06 23:56:46.798453 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:46.798434 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:46.813530 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:46.813506 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:46.841599 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:46.841555 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" podStartSLOduration=7.946642976 podStartE2EDuration="28.841539782s" totalImagesPullingTime="20.894896806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:46.841317228 +0000 UTC m=+29.981774285" watchObservedRunningTime="2026-07-06 23:56:46.841539782 +0000 UTC m=+29.981996837" Jul 06 23:56:47.481334 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:47.481169 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:47.481538 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:47.481214 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:47.481538 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:47.481397 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:47.481538 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:47.481483 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:47.800742 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:47.800652 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:47.801107 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:47.800745 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:47.814952 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:47.814924 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:56:47.875866 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:47.875836 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bcn92"] Jul 06 23:56:47.876011 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:47.875943 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:47.876063 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:47.876029 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:47.883572 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:47.883504 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x5k59"] Jul 06 23:56:47.883696 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:47.883621 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:47.883759 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:47.883718 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:47.884207 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:47.884186 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mddqt"] Jul 06 23:56:47.884309 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:47.884287 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:47.884446 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:47.884388 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:48.805093 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:48.805063 2568 generic.go:356] "Generic (PLEG): container finished" podID="5ca7a76e-7c60-4e0c-bbda-a1edb3def89d" containerID="439e4e01226a2ff5212ad23481c516502223121bd43cfa40c8c2ca6388308c16" exitCode=0 Jul 06 23:56:48.805546 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:48.805108 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlz2q" event={"ID":"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d","Type":"ContainerDied","Data":"439e4e01226a2ff5212ad23481c516502223121bd43cfa40c8c2ca6388308c16"} Jul 06 23:56:49.481258 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:49.481231 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:49.481409 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:49.481231 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:49.481409 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:49.481356 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:49.481409 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:49.481234 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:49.481555 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:49.481409 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:49.481555 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:49.481485 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:50.811120 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:50.811082 2568 generic.go:356] "Generic (PLEG): container finished" podID="5ca7a76e-7c60-4e0c-bbda-a1edb3def89d" containerID="93e7f592b62c5fe6d2c2a88049cae9f084e9b46c880433b00051b3afc14b4973" exitCode=0 Jul 06 23:56:50.811501 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:50.811119 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlz2q" event={"ID":"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d","Type":"ContainerDied","Data":"93e7f592b62c5fe6d2c2a88049cae9f084e9b46c880433b00051b3afc14b4973"} Jul 06 23:56:51.091991 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:51.091919 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:51.092121 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:51.092063 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:51.092160 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:51.092125 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret podName:76bea87f-a057-4a59-8e4a-171542fa018a nodeName:}" failed. No retries permitted until 2026-07-06 23:57:23.092109207 +0000 UTC m=+66.232566241 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret") pod "global-pull-secret-syncer-x5k59" (UID: "76bea87f-a057-4a59-8e4a-171542fa018a") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:51.193144 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:51.193107 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs\") pod \"network-metrics-daemon-mddqt\" (UID: \"b8d4e21f-d7f0-404d-a8e8-4da642d240c7\") " pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:51.193297 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:51.193151 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55bq\" (UniqueName: \"kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq\") pod \"network-check-target-bcn92\" (UID: \"388db162-a514-4da6-b4ad-398a94e22e01\") " pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:51.193297 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:51.193249 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:51.193383 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:51.193311 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs podName:b8d4e21f-d7f0-404d-a8e8-4da642d240c7 nodeName:}" failed. No retries permitted until 2026-07-06 23:57:23.193296133 +0000 UTC m=+66.333753168 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs") pod "network-metrics-daemon-mddqt" (UID: "b8d4e21f-d7f0-404d-a8e8-4da642d240c7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:51.193383 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:51.193257 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:51.193383 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:51.193354 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:51.193383 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:51.193366 2568 projected.go:196] Error preparing data for projected volume kube-api-access-k55bq for pod openshift-network-diagnostics/network-check-target-bcn92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:51.193537 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:51.193413 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq podName:388db162-a514-4da6-b4ad-398a94e22e01 nodeName:}" failed. No retries permitted until 2026-07-06 23:57:23.193400207 +0000 UTC m=+66.333857259 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k55bq" (UniqueName: "kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq") pod "network-check-target-bcn92" (UID: "388db162-a514-4da6-b4ad-398a94e22e01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:51.480942 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:51.480913 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:51.480942 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:51.480942 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:51.481190 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:51.480922 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:51.481190 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:51.481039 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:51.481190 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:51.481141 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:51.481335 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:51.481208 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:53.480748 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:53.480712 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:53.481220 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:53.480723 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:53.481220 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:53.480848 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:53.481220 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:53.480848 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5k59" podUID="76bea87f-a057-4a59-8e4a-171542fa018a" Jul 06 23:56:53.481220 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:53.480968 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcn92" podUID="388db162-a514-4da6-b4ad-398a94e22e01" Jul 06 23:56:53.481220 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:53.481088 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mddqt" podUID="b8d4e21f-d7f0-404d-a8e8-4da642d240c7" Jul 06 23:56:54.143015 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.142819 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeReady" Jul 06 23:56:54.143185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.143127 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 23:56:54.178259 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.178208 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d47887bfc-2r99h"] Jul 06 23:56:54.183673 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.183645 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.189006 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.188963 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 23:56:54.189158 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.189143 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 23:56:54.189217 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.189169 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hbkmd\"" Jul 06 23:56:54.189217 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.189180 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 23:56:54.191703 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.191671 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4txr6"] Jul 06 23:56:54.196019 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.195972 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:54.196328 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.196305 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 23:56:54.197259 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.197235 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d47887bfc-2r99h"] Jul 06 23:56:54.198466 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.198410 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 23:56:54.198571 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.198507 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 23:56:54.198571 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.198507 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jf85g\"" Jul 06 23:56:54.200055 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.200033 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fmpf9"] Jul 06 23:56:54.206302 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.206238 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4txr6"] Jul 06 23:56:54.206471 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.206436 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fmpf9" Jul 06 23:56:54.208998 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.208973 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 23:56:54.209562 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.209255 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 23:56:54.209562 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.209255 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xpwtc\"" Jul 06 23:56:54.209562 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.209344 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 23:56:54.219000 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.218896 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fmpf9"] Jul 06 23:56:54.316347 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.316257 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d482c\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-kube-api-access-d482c\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.316347 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.316316 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b80472f8-19c1-4b5b-b721-b611a82cc28c-cert\") pod \"ingress-canary-fmpf9\" (UID: \"b80472f8-19c1-4b5b-b721-b611a82cc28c\") " pod="openshift-ingress-canary/ingress-canary-fmpf9" Jul 06 23:56:54.316586 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.316390 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-ca-trust-extracted\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.316586 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.316498 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:54.316586 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.316538 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-installation-pull-secrets\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.316586 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.316576 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-trusted-ca\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.316835 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.316598 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nrgn\" (UniqueName: \"kubernetes.io/projected/b80472f8-19c1-4b5b-b721-b611a82cc28c-kube-api-access-8nrgn\") pod \"ingress-canary-fmpf9\" (UID: \"b80472f8-19c1-4b5b-b721-b611a82cc28c\") " pod="openshift-ingress-canary/ingress-canary-fmpf9" Jul 06 23:56:54.316835 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.316620 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6e160535-0c07-49d0-b690-18ca6aee0d87-config-volume\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:54.316835 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.316643 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.316835 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.316685 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-certificates\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.316835 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.316746 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e160535-0c07-49d0-b690-18ca6aee0d87-tmp-dir\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:54.317047 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.316833 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-bound-sa-token\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.317047 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.316898 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hbdl\" (UniqueName: \"kubernetes.io/projected/6e160535-0c07-49d0-b690-18ca6aee0d87-kube-api-access-4hbdl\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:54.317047 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.316931 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-image-registry-private-configuration\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.417861 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.417816 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-trusted-ca\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.417861 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.417866 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nrgn\" (UniqueName: \"kubernetes.io/projected/b80472f8-19c1-4b5b-b721-b611a82cc28c-kube-api-access-8nrgn\") pod \"ingress-canary-fmpf9\" (UID: \"b80472f8-19c1-4b5b-b721-b611a82cc28c\") " pod="openshift-ingress-canary/ingress-canary-fmpf9" Jul 06 23:56:54.418209 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.417899 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6e160535-0c07-49d0-b690-18ca6aee0d87-config-volume\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:54.418209 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.417927 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.418209 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.417952 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-certificates\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.418209 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.417979 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e160535-0c07-49d0-b690-18ca6aee0d87-tmp-dir\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:54.418209 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:54.418091 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:56:54.418209 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:54.418112 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d47887bfc-2r99h: secret "image-registry-tls" not found Jul 06 23:56:54.418209 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:54.418171 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls podName:e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c nodeName:}" failed. No retries permitted until 2026-07-06 23:56:54.918149938 +0000 UTC m=+38.058606980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls") pod "image-registry-5d47887bfc-2r99h" (UID: "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c") : secret "image-registry-tls" not found Jul 06 23:56:54.418209 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.418194 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-bound-sa-token\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.418650 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.418234 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hbdl\" (UniqueName: \"kubernetes.io/projected/6e160535-0c07-49d0-b690-18ca6aee0d87-kube-api-access-4hbdl\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:54.418650 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.418265 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-image-registry-private-configuration\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.418650 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.418307 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d482c\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-kube-api-access-d482c\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.418650 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.418333 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b80472f8-19c1-4b5b-b721-b611a82cc28c-cert\") pod \"ingress-canary-fmpf9\" (UID: \"b80472f8-19c1-4b5b-b721-b611a82cc28c\") " pod="openshift-ingress-canary/ingress-canary-fmpf9" Jul 06 23:56:54.418650 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.418352 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e160535-0c07-49d0-b690-18ca6aee0d87-tmp-dir\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:54.418650 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.418358 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-ca-trust-extracted\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.418650 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.418466 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:54.418650 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.418499 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-installation-pull-secrets\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.418650 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:54.418642 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:56:54.419072 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:54.418690 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls podName:6e160535-0c07-49d0-b690-18ca6aee0d87 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:54.918674106 +0000 UTC m=+38.059131141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls") pod "dns-default-4txr6" (UID: "6e160535-0c07-49d0-b690-18ca6aee0d87") : secret "dns-default-metrics-tls" not found Jul 06 23:56:54.419072 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.418701 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-ca-trust-extracted\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.419072 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.419039 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-trusted-ca\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.419189 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.419086 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-certificates\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.419465 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.419448 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6e160535-0c07-49d0-b690-18ca6aee0d87-config-volume\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:54.423228 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.423203 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-installation-pull-secrets\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.423228 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.423218 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b80472f8-19c1-4b5b-b721-b611a82cc28c-cert\") pod \"ingress-canary-fmpf9\" (UID: \"b80472f8-19c1-4b5b-b721-b611a82cc28c\") " pod="openshift-ingress-canary/ingress-canary-fmpf9" Jul 06 23:56:54.423389 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.423203 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-image-registry-private-configuration\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.432525 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.432473 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hbdl\" (UniqueName: \"kubernetes.io/projected/6e160535-0c07-49d0-b690-18ca6aee0d87-kube-api-access-4hbdl\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:54.432852 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.432828 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-bound-sa-token\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.434336 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.434312 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d482c\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-kube-api-access-d482c\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.434479 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.434461 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nrgn\" (UniqueName: \"kubernetes.io/projected/b80472f8-19c1-4b5b-b721-b611a82cc28c-kube-api-access-8nrgn\") pod \"ingress-canary-fmpf9\" (UID: \"b80472f8-19c1-4b5b-b721-b611a82cc28c\") " pod="openshift-ingress-canary/ingress-canary-fmpf9" Jul 06 23:56:54.518363 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.518327 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fmpf9" Jul 06 23:56:54.858561 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.858534 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fmpf9"] Jul 06 23:56:54.861558 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:54.861534 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80472f8_19c1_4b5b_b721_b611a82cc28c.slice/crio-30c82d7971fa68e9d919fe2db5435a619578f37dc20a3d18e8e2cbdf472f6561 WatchSource:0}: Error finding container 30c82d7971fa68e9d919fe2db5435a619578f37dc20a3d18e8e2cbdf472f6561: Status 404 returned error can't find the container with id 30c82d7971fa68e9d919fe2db5435a619578f37dc20a3d18e8e2cbdf472f6561 Jul 06 23:56:54.923934 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.923693 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:54.924085 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:54.923963 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:54.924085 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:54.923848 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:56:54.924085 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:54.924047 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls podName:6e160535-0c07-49d0-b690-18ca6aee0d87 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:55.924026964 +0000 UTC m=+39.064484018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls") pod "dns-default-4txr6" (UID: "6e160535-0c07-49d0-b690-18ca6aee0d87") : secret "dns-default-metrics-tls" not found Jul 06 23:56:54.924282 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:54.924101 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:56:54.924282 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:54.924110 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d47887bfc-2r99h: secret "image-registry-tls" not found Jul 06 23:56:54.924282 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:54.924143 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls podName:e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c nodeName:}" failed. No retries permitted until 2026-07-06 23:56:55.924135721 +0000 UTC m=+39.064592760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls") pod "image-registry-5d47887bfc-2r99h" (UID: "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c") : secret "image-registry-tls" not found Jul 06 23:56:55.480524 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.480489 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:56:55.480688 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.480491 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:56:55.480743 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.480678 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:56:55.484537 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.484516 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 23:56:55.484671 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.484648 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 23:56:55.485356 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.485314 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lswch\"" Jul 06 23:56:55.485475 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.485409 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-knf8z\"" Jul 06 23:56:55.485475 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.485433 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 23:56:55.485600 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.485539 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 23:56:55.825627 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.825545 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fmpf9" event={"ID":"b80472f8-19c1-4b5b-b721-b611a82cc28c","Type":"ContainerStarted","Data":"30c82d7971fa68e9d919fe2db5435a619578f37dc20a3d18e8e2cbdf472f6561"} Jul 06 23:56:55.828861 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.828831 2568 generic.go:356] "Generic (PLEG): container finished" podID="5ca7a76e-7c60-4e0c-bbda-a1edb3def89d" containerID="ae066c2c8a9353834631036ccf07c64aaee7d793901c78fba17983441b11926a" exitCode=0 Jul 06 23:56:55.828981 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.828894 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlz2q" event={"ID":"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d","Type":"ContainerDied","Data":"ae066c2c8a9353834631036ccf07c64aaee7d793901c78fba17983441b11926a"} Jul 06 23:56:55.933733 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.933701 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:55.934436 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.933860 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:55.934436 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:55.934207 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:56:55.934436 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:55.934228 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d47887bfc-2r99h: secret "image-registry-tls" not found Jul 06 23:56:55.934436 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:55.934281 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls podName:e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c nodeName:}" failed. No retries permitted until 2026-07-06 23:56:57.934261979 +0000 UTC m=+41.074719035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls") pod "image-registry-5d47887bfc-2r99h" (UID: "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c") : secret "image-registry-tls" not found Jul 06 23:56:55.934436 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:55.934359 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:56:55.934436 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:55.934395 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls podName:6e160535-0c07-49d0-b690-18ca6aee0d87 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:57.934381411 +0000 UTC m=+41.074838446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls") pod "dns-default-4txr6" (UID: "6e160535-0c07-49d0-b690-18ca6aee0d87") : secret "dns-default-metrics-tls" not found Jul 06 23:56:55.975057 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.975023 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bbg4s"] Jul 06 23:56:55.978833 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.978809 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:55.983859 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.983840 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 23:56:55.984042 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.984019 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 23:56:55.984220 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.984202 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q9bzd\"" Jul 06 23:56:55.984277 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.984256 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 23:56:55.988497 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:55.988477 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 23:56:56.015812 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.015776 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bbg4s"] Jul 06 23:56:56.136238 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.136151 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/47bf1021-e8fc-4b54-b1b3-de741519d372-crio-socket\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.136238 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.136188 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/47bf1021-e8fc-4b54-b1b3-de741519d372-data-volume\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.136238 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.136211 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.136578 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.136328 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.136578 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.136389 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2hjk\" (UniqueName: \"kubernetes.io/projected/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-api-access-t2hjk\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.237621 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.237578 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/47bf1021-e8fc-4b54-b1b3-de741519d372-data-volume\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.237820 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.237637 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.237820 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.237708 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.237820 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.237760 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2hjk\" (UniqueName: \"kubernetes.io/projected/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-api-access-t2hjk\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.238084 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:56.237853 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:56:56.238084 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:56.237885 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:56:56.238084 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:56.237938 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm podName:47bf1021-e8fc-4b54-b1b3-de741519d372 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:56.737919097 +0000 UTC m=+39.878376151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bbg4s" (UID: "47bf1021-e8fc-4b54-b1b3-de741519d372") : configmap "kube-rbac-proxy" not found Jul 06 23:56:56.238084 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:56.237993 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls podName:47bf1021-e8fc-4b54-b1b3-de741519d372 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:56.737972974 +0000 UTC m=+39.878430023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bbg4s" (UID: "47bf1021-e8fc-4b54-b1b3-de741519d372") : secret "insights-runtime-extractor-tls" not found Jul 06 23:56:56.238084 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.238008 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/47bf1021-e8fc-4b54-b1b3-de741519d372-data-volume\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.238084 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.238027 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/47bf1021-e8fc-4b54-b1b3-de741519d372-crio-socket\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.238385 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.238267 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/47bf1021-e8fc-4b54-b1b3-de741519d372-crio-socket\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.238847 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.238827 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-2nmxh"] Jul 06 23:56:56.243622 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.243603 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2nmxh" Jul 06 23:56:56.246155 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.246126 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 23:56:56.246273 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.246217 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 23:56:56.246548 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.246507 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-d9n9z\"" Jul 06 23:56:56.258825 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.258753 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-2nmxh"] Jul 06 23:56:56.261561 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.261534 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2hjk\" (UniqueName: \"kubernetes.io/projected/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-api-access-t2hjk\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.338595 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.338556 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs2kx\" (UniqueName: \"kubernetes.io/projected/3c348d75-7127-444e-9da4-3a8845d0a8a9-kube-api-access-gs2kx\") pod \"migrator-d57d58cc9-2nmxh\" (UID: \"3c348d75-7127-444e-9da4-3a8845d0a8a9\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2nmxh" Jul 06 23:56:56.439408 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.439376 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs2kx\" (UniqueName: \"kubernetes.io/projected/3c348d75-7127-444e-9da4-3a8845d0a8a9-kube-api-access-gs2kx\") pod \"migrator-d57d58cc9-2nmxh\" (UID: \"3c348d75-7127-444e-9da4-3a8845d0a8a9\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2nmxh" Jul 06 23:56:56.466349 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.466319 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs2kx\" (UniqueName: \"kubernetes.io/projected/3c348d75-7127-444e-9da4-3a8845d0a8a9-kube-api-access-gs2kx\") pod \"migrator-d57d58cc9-2nmxh\" (UID: \"3c348d75-7127-444e-9da4-3a8845d0a8a9\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2nmxh" Jul 06 23:56:56.554153 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.554119 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2nmxh" Jul 06 23:56:56.741838 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.741750 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.741838 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.741821 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:56.742050 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:56.741867 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:56:56.742050 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:56.741926 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:56:56.742050 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:56.741941 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls podName:47bf1021-e8fc-4b54-b1b3-de741519d372 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:57.741921342 +0000 UTC m=+40.882378391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bbg4s" (UID: "47bf1021-e8fc-4b54-b1b3-de741519d372") : secret "insights-runtime-extractor-tls" not found Jul 06 23:56:56.742050 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:56.741991 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm podName:47bf1021-e8fc-4b54-b1b3-de741519d372 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:57.741973352 +0000 UTC m=+40.882430405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bbg4s" (UID: "47bf1021-e8fc-4b54-b1b3-de741519d372") : configmap "kube-rbac-proxy" not found Jul 06 23:56:56.835587 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.835555 2568 generic.go:356] "Generic (PLEG): container finished" podID="5ca7a76e-7c60-4e0c-bbda-a1edb3def89d" containerID="8c9589ed0f323ecbe3f9a78b58002fdb99b9f2a8dc24b78b42751553cb006a25" exitCode=0 Jul 06 23:56:56.836026 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:56.835590 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlz2q" event={"ID":"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d","Type":"ContainerDied","Data":"8c9589ed0f323ecbe3f9a78b58002fdb99b9f2a8dc24b78b42751553cb006a25"} Jul 06 23:56:57.048716 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:57.048534 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-2nmxh"] Jul 06 23:56:57.156349 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:57.156321 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c348d75_7127_444e_9da4_3a8845d0a8a9.slice/crio-e443890ccd610a6b58168131114eedef1902246157f774fe0f91a0556935bc04 WatchSource:0}: Error finding container e443890ccd610a6b58168131114eedef1902246157f774fe0f91a0556935bc04: Status 404 returned error can't find the container with id e443890ccd610a6b58168131114eedef1902246157f774fe0f91a0556935bc04 Jul 06 23:56:57.750014 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:57.749970 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:57.750197 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:57.750053 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:57.750197 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:57.750143 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:56:57.750197 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:57.750168 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:56:57.750354 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:57.750214 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls podName:47bf1021-e8fc-4b54-b1b3-de741519d372 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:59.750194131 +0000 UTC m=+42.890651169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bbg4s" (UID: "47bf1021-e8fc-4b54-b1b3-de741519d372") : secret "insights-runtime-extractor-tls" not found Jul 06 23:56:57.750354 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:57.750233 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm podName:47bf1021-e8fc-4b54-b1b3-de741519d372 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:59.750225522 +0000 UTC m=+42.890682556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bbg4s" (UID: "47bf1021-e8fc-4b54-b1b3-de741519d372") : configmap "kube-rbac-proxy" not found Jul 06 23:56:57.842009 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:57.841977 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlz2q" event={"ID":"5ca7a76e-7c60-4e0c-bbda-a1edb3def89d","Type":"ContainerStarted","Data":"783dac8ab6765ab42b451651cae32f1de76b913d7d4f2ae96a84035bda0cccb3"} Jul 06 23:56:57.843521 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:57.843488 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fmpf9" event={"ID":"b80472f8-19c1-4b5b-b721-b611a82cc28c","Type":"ContainerStarted","Data":"524a8427f718fdcb4300b0d16b16269d322ba1f04e950c31ff86fd15a7214df9"} Jul 06 23:56:57.844627 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:57.844603 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2nmxh" event={"ID":"3c348d75-7127-444e-9da4-3a8845d0a8a9","Type":"ContainerStarted","Data":"e443890ccd610a6b58168131114eedef1902246157f774fe0f91a0556935bc04"} Jul 06 23:56:57.855766 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:57.855746 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s8cmz_e3979fa5-6b05-4e90-abf0-ae3cb0286f0c/dns-node-resolver/0.log" Jul 06 23:56:57.870119 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:57.870083 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qlz2q" podStartSLOduration=8.791587883 podStartE2EDuration="39.87007142s" totalImagesPullingTime="31.078483537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:57.869391066 +0000 UTC m=+41.009848123" watchObservedRunningTime="2026-07-06 23:56:57.87007142 +0000 UTC m=+41.010528484" Jul 06 23:56:57.887740 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:57.887689 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fmpf9" podStartSLOduration=1.571075478 podStartE2EDuration="3.887671834s" totalImagesPullingTime="2.316596356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:57.887099666 +0000 UTC m=+41.027556721" watchObservedRunningTime="2026-07-06 23:56:57.887671834 +0000 UTC m=+41.028128891" Jul 06 23:56:57.951824 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:57.951794 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:56:57.951978 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:57.951905 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:56:57.951978 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:57.951917 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d47887bfc-2r99h: secret "image-registry-tls" not found Jul 06 23:56:57.951978 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:57.951959 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls podName:e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c nodeName:}" failed. No retries permitted until 2026-07-06 23:57:01.951945299 +0000 UTC m=+45.092402362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls") pod "image-registry-5d47887bfc-2r99h" (UID: "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c") : secret "image-registry-tls" not found Jul 06 23:56:57.952131 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:57.952046 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:56:57.952258 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:57.952234 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:56:57.952377 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:57.952309 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls podName:6e160535-0c07-49d0-b690-18ca6aee0d87 nodeName:}" failed. No retries permitted until 2026-07-06 23:57:01.95227686 +0000 UTC m=+45.092733900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls") pod "dns-default-4txr6" (UID: "6e160535-0c07-49d0-b690-18ca6aee0d87") : secret "dns-default-metrics-tls" not found Jul 06 23:56:58.184881 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.184799 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-zspjm"] Jul 06 23:56:58.198337 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.198306 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-zspjm"] Jul 06 23:56:58.198522 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.198342 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-zspjm" Jul 06 23:56:58.201408 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.201386 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 06 23:56:58.201408 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.201401 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jvcsh\"" Jul 06 23:56:58.201593 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.201483 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 06 23:56:58.201774 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.201755 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 06 23:56:58.201854 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.201816 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 06 23:56:58.355865 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.355835 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fd607da7-f719-4d75-89a6-2439c1b25dc1-signing-cabundle\") pod \"service-ca-758f599c5-zspjm\" (UID: \"fd607da7-f719-4d75-89a6-2439c1b25dc1\") " pod="openshift-service-ca/service-ca-758f599c5-zspjm" Jul 06 23:56:58.356008 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.355900 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fd607da7-f719-4d75-89a6-2439c1b25dc1-signing-key\") pod \"service-ca-758f599c5-zspjm\" (UID: \"fd607da7-f719-4d75-89a6-2439c1b25dc1\") " pod="openshift-service-ca/service-ca-758f599c5-zspjm" Jul 06 23:56:58.356057 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.356018 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbmxp\" (UniqueName: \"kubernetes.io/projected/fd607da7-f719-4d75-89a6-2439c1b25dc1-kube-api-access-jbmxp\") pod \"service-ca-758f599c5-zspjm\" (UID: \"fd607da7-f719-4d75-89a6-2439c1b25dc1\") " pod="openshift-service-ca/service-ca-758f599c5-zspjm" Jul 06 23:56:58.455859 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.455836 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4t5gn_56059632-3851-4ea0-8a86-fcabfc509e65/node-ca/0.log" Jul 06 23:56:58.457048 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.457025 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fd607da7-f719-4d75-89a6-2439c1b25dc1-signing-key\") pod \"service-ca-758f599c5-zspjm\" (UID: \"fd607da7-f719-4d75-89a6-2439c1b25dc1\") " pod="openshift-service-ca/service-ca-758f599c5-zspjm" Jul 06 23:56:58.457165 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.457114 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbmxp\" (UniqueName: \"kubernetes.io/projected/fd607da7-f719-4d75-89a6-2439c1b25dc1-kube-api-access-jbmxp\") pod \"service-ca-758f599c5-zspjm\" (UID: \"fd607da7-f719-4d75-89a6-2439c1b25dc1\") " pod="openshift-service-ca/service-ca-758f599c5-zspjm" Jul 06 23:56:58.457165 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.457136 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fd607da7-f719-4d75-89a6-2439c1b25dc1-signing-cabundle\") pod \"service-ca-758f599c5-zspjm\" (UID: \"fd607da7-f719-4d75-89a6-2439c1b25dc1\") " pod="openshift-service-ca/service-ca-758f599c5-zspjm" Jul 06 23:56:58.457759 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.457743 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fd607da7-f719-4d75-89a6-2439c1b25dc1-signing-cabundle\") pod \"service-ca-758f599c5-zspjm\" (UID: \"fd607da7-f719-4d75-89a6-2439c1b25dc1\") " pod="openshift-service-ca/service-ca-758f599c5-zspjm" Jul 06 23:56:58.460599 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.460578 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fd607da7-f719-4d75-89a6-2439c1b25dc1-signing-key\") pod \"service-ca-758f599c5-zspjm\" (UID: \"fd607da7-f719-4d75-89a6-2439c1b25dc1\") " pod="openshift-service-ca/service-ca-758f599c5-zspjm" Jul 06 23:56:58.467387 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.467364 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbmxp\" (UniqueName: \"kubernetes.io/projected/fd607da7-f719-4d75-89a6-2439c1b25dc1-kube-api-access-jbmxp\") pod \"service-ca-758f599c5-zspjm\" (UID: \"fd607da7-f719-4d75-89a6-2439c1b25dc1\") " pod="openshift-service-ca/service-ca-758f599c5-zspjm" Jul 06 23:56:58.508558 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.508537 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-zspjm" Jul 06 23:56:58.721729 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.721439 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-zspjm"] Jul 06 23:56:58.726455 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:56:58.726407 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd607da7_f719_4d75_89a6_2439c1b25dc1.slice/crio-880a2c646cc9c241e9d1bdb0547aed58fe5a20a73b4967e22c008e1657cdd00a WatchSource:0}: Error finding container 880a2c646cc9c241e9d1bdb0547aed58fe5a20a73b4967e22c008e1657cdd00a: Status 404 returned error can't find the container with id 880a2c646cc9c241e9d1bdb0547aed58fe5a20a73b4967e22c008e1657cdd00a Jul 06 23:56:58.848861 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.848831 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-zspjm" event={"ID":"fd607da7-f719-4d75-89a6-2439c1b25dc1","Type":"ContainerStarted","Data":"880a2c646cc9c241e9d1bdb0547aed58fe5a20a73b4967e22c008e1657cdd00a"} Jul 06 23:56:58.850928 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.850900 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2nmxh" event={"ID":"3c348d75-7127-444e-9da4-3a8845d0a8a9","Type":"ContainerStarted","Data":"031858a23b2c28fff24f9e90e475e91dd2910ae7bbe9e72f4275191afc440ff1"} Jul 06 23:56:58.851041 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.850957 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2nmxh" event={"ID":"3c348d75-7127-444e-9da4-3a8845d0a8a9","Type":"ContainerStarted","Data":"f8721ac3bafa0798b5a87b35589c336783bdb3af01af17c3630de36b86a529be"} Jul 06 23:56:58.871917 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:58.871869 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2nmxh" podStartSLOduration=1.454691161 podStartE2EDuration="2.871852693s" totalImagesPullingTime="1.417161532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:58.867465574 +0000 UTC m=+42.007922630" watchObservedRunningTime="2026-07-06 23:56:58.871852693 +0000 UTC m=+42.012309749" Jul 06 23:56:59.255553 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:59.255527 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fmpf9_b80472f8-19c1-4b5b-b721-b611a82cc28c/serve-healthcheck-canary/0.log" Jul 06 23:56:59.767715 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:59.767679 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:59.767881 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:59.767840 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:56:59.767942 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:56:59.767878 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:56:59.767942 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:59.767910 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls podName:47bf1021-e8fc-4b54-b1b3-de741519d372 nodeName:}" failed. No retries permitted until 2026-07-06 23:57:03.767893671 +0000 UTC m=+46.908350705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bbg4s" (UID: "47bf1021-e8fc-4b54-b1b3-de741519d372") : secret "insights-runtime-extractor-tls" not found Jul 06 23:56:59.768063 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:59.767948 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:56:59.768063 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:56:59.767997 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm podName:47bf1021-e8fc-4b54-b1b3-de741519d372 nodeName:}" failed. No retries permitted until 2026-07-06 23:57:03.76798134 +0000 UTC m=+46.908438390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bbg4s" (UID: "47bf1021-e8fc-4b54-b1b3-de741519d372") : configmap "kube-rbac-proxy" not found Jul 06 23:57:01.860239 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:01.860205 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-zspjm" event={"ID":"fd607da7-f719-4d75-89a6-2439c1b25dc1","Type":"ContainerStarted","Data":"9c03544b6c814969487169829140df39b9788728415fe9435c9d73925a478d22"} Jul 06 23:57:01.888607 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:01.888557 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-zspjm" podStartSLOduration=1.8134260580000001 podStartE2EDuration="3.888538644s" totalImagesPullingTime="2.075112586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:01.887720167 +0000 UTC m=+45.028177219" watchObservedRunningTime="2026-07-06 23:57:01.888538644 +0000 UTC m=+45.028995695" Jul 06 23:57:01.981441 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:01.981392 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:57:01.981601 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:01.981484 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:57:01.981601 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:57:01.981514 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:57:01.981601 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:57:01.981575 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls podName:6e160535-0c07-49d0-b690-18ca6aee0d87 nodeName:}" failed. No retries permitted until 2026-07-06 23:57:09.981560268 +0000 UTC m=+53.122017306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls") pod "dns-default-4txr6" (UID: "6e160535-0c07-49d0-b690-18ca6aee0d87") : secret "dns-default-metrics-tls" not found Jul 06 23:57:01.981750 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:57:01.981619 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:57:01.981750 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:57:01.981636 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d47887bfc-2r99h: secret "image-registry-tls" not found Jul 06 23:57:01.981750 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:57:01.981682 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls podName:e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c nodeName:}" failed. No retries permitted until 2026-07-06 23:57:09.981668614 +0000 UTC m=+53.122125647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls") pod "image-registry-5d47887bfc-2r99h" (UID: "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c") : secret "image-registry-tls" not found Jul 06 23:57:03.793612 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:03.793567 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:57:03.793999 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:03.793652 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:57:03.793999 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:57:03.793750 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:57:03.793999 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:57:03.793802 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls podName:47bf1021-e8fc-4b54-b1b3-de741519d372 nodeName:}" failed. No retries permitted until 2026-07-06 23:57:11.793788653 +0000 UTC m=+54.934245688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bbg4s" (UID: "47bf1021-e8fc-4b54-b1b3-de741519d372") : secret "insights-runtime-extractor-tls" not found Jul 06 23:57:03.794175 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:03.794153 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/47bf1021-e8fc-4b54-b1b3-de741519d372-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:57:10.043467 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:10.043405 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:57:10.044027 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:10.043606 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:57:10.046581 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:10.046559 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e160535-0c07-49d0-b690-18ca6aee0d87-metrics-tls\") pod \"dns-default-4txr6\" (UID: \"6e160535-0c07-49d0-b690-18ca6aee0d87\") " pod="openshift-dns/dns-default-4txr6" Jul 06 23:57:10.046687 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:10.046650 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls\") pod \"image-registry-5d47887bfc-2r99h\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:57:10.098515 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:10.098476 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:57:10.111319 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:10.111291 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4txr6" Jul 06 23:57:10.269335 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:10.269286 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d47887bfc-2r99h"] Jul 06 23:57:10.272450 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:57:10.272395 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7976dfa_a4b1_46b6_b0f1_ac2a7e95454c.slice/crio-41c8f19a4c51457d35e720b48a33d04e7afe334d79ee52829e622172d8a4df7b WatchSource:0}: Error finding container 41c8f19a4c51457d35e720b48a33d04e7afe334d79ee52829e622172d8a4df7b: Status 404 returned error can't find the container with id 41c8f19a4c51457d35e720b48a33d04e7afe334d79ee52829e622172d8a4df7b Jul 06 23:57:10.280752 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:10.280544 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4txr6"] Jul 06 23:57:10.284740 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:57:10.284718 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e160535_0c07_49d0_b690_18ca6aee0d87.slice/crio-293a86a1bcfeaf064f278e05f8c7e6e6e740afd6624d5225a7e64fe0810c7d9f WatchSource:0}: Error finding container 293a86a1bcfeaf064f278e05f8c7e6e6e740afd6624d5225a7e64fe0810c7d9f: Status 404 returned error can't find the container with id 293a86a1bcfeaf064f278e05f8c7e6e6e740afd6624d5225a7e64fe0810c7d9f Jul 06 23:57:10.885218 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:10.885135 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" event={"ID":"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c","Type":"ContainerStarted","Data":"d604bb738a467e96ba439523614ef3cdab3d8224295d5fb008d9c3bef455b513"} Jul 06 23:57:10.885218 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:10.885204 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:57:10.885218 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:10.885224 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" event={"ID":"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c","Type":"ContainerStarted","Data":"41c8f19a4c51457d35e720b48a33d04e7afe334d79ee52829e622172d8a4df7b"} Jul 06 23:57:10.886754 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:10.886728 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4txr6" event={"ID":"6e160535-0c07-49d0-b690-18ca6aee0d87","Type":"ContainerStarted","Data":"293a86a1bcfeaf064f278e05f8c7e6e6e740afd6624d5225a7e64fe0810c7d9f"} Jul 06 23:57:10.907238 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:10.907181 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" podStartSLOduration=23.907159941 podStartE2EDuration="23.907159941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:10.906141029 +0000 UTC m=+54.046598085" watchObservedRunningTime="2026-07-06 23:57:10.907159941 +0000 UTC m=+54.047616999" Jul 06 23:57:11.859343 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:11.859305 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:57:11.861543 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:11.861522 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/47bf1021-e8fc-4b54-b1b3-de741519d372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbg4s\" (UID: \"47bf1021-e8fc-4b54-b1b3-de741519d372\") " pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:57:11.906846 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:11.906812 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bbg4s" Jul 06 23:57:12.038577 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:12.038547 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bbg4s"] Jul 06 23:57:12.192490 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:57:12.192461 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47bf1021_e8fc_4b54_b1b3_de741519d372.slice/crio-2346d64552832860894162aebefd24d5aad92097757a2e88e84da1cfdbe16057 WatchSource:0}: Error finding container 2346d64552832860894162aebefd24d5aad92097757a2e88e84da1cfdbe16057: Status 404 returned error can't find the container with id 2346d64552832860894162aebefd24d5aad92097757a2e88e84da1cfdbe16057 Jul 06 23:57:12.898032 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:12.897930 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4txr6" event={"ID":"6e160535-0c07-49d0-b690-18ca6aee0d87","Type":"ContainerStarted","Data":"adb44cfcb8eed692aa46e5446c495263e5d576c4c5ff7494848d664f39707dfa"} Jul 06 23:57:12.898032 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:12.898007 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4txr6" event={"ID":"6e160535-0c07-49d0-b690-18ca6aee0d87","Type":"ContainerStarted","Data":"628d4f1990333e4358e2730eb50b0bb9b8cc1eba81f0dced71f5dc1dcd3813b5"} Jul 06 23:57:12.898574 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:12.898041 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4txr6" Jul 06 23:57:12.899708 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:12.899643 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bbg4s" event={"ID":"47bf1021-e8fc-4b54-b1b3-de741519d372","Type":"ContainerStarted","Data":"2346d64552832860894162aebefd24d5aad92097757a2e88e84da1cfdbe16057"} Jul 06 23:57:12.925627 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:12.925527 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4txr6" podStartSLOduration=16.987034242 podStartE2EDuration="18.925504155s" totalImagesPullingTime="1.938469913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:12.924983543 +0000 UTC m=+56.065440622" watchObservedRunningTime="2026-07-06 23:57:12.925504155 +0000 UTC m=+56.065961208" Jul 06 23:57:15.912787 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:15.912755 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bbg4s" event={"ID":"47bf1021-e8fc-4b54-b1b3-de741519d372","Type":"ContainerStarted","Data":"5c434e9fd429daea78607ed5d651f48a73a19ada9bbe284d462b0ad8a3de2039"} Jul 06 23:57:16.917686 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:16.917603 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bbg4s" event={"ID":"47bf1021-e8fc-4b54-b1b3-de741519d372","Type":"ContainerStarted","Data":"b5cc13560824cddfca263031ae441330dc01eec0781af52d2cb39c45c4188e13"} Jul 06 23:57:19.822047 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:19.821970 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rg8hd" Jul 06 23:57:19.929185 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:19.929147 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bbg4s" event={"ID":"47bf1021-e8fc-4b54-b1b3-de741519d372","Type":"ContainerStarted","Data":"0f232540db4e0488819278238c8a20044915642d5ffecae522cdbf9b7e63c012"} Jul 06 23:57:19.947096 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:19.947050 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bbg4s" podStartSLOduration=17.759837186 podStartE2EDuration="24.947035855s" totalImagesPullingTime="7.187198669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:19.946338199 +0000 UTC m=+63.086795255" watchObservedRunningTime="2026-07-06 23:57:19.947035855 +0000 UTC m=+63.087492910" Jul 06 23:57:20.924123 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.924089 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6"] Jul 06 23:57:20.926312 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.926297 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8"] Jul 06 23:57:20.926447 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.926415 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6" Jul 06 23:57:20.928117 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.928097 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" Jul 06 23:57:20.929498 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.929414 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 23:57:20.929605 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.929567 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-vnm2h\"" Jul 06 23:57:20.929723 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.929708 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 06 23:57:20.929998 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.929983 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 23:57:20.932516 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.932501 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-xxxjj"] Jul 06 23:57:20.933612 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.933345 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 23:57:20.934233 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.934217 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xxxjj" Jul 06 23:57:20.934640 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.934626 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 23:57:20.936478 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.936463 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 23:57:20.936662 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.936647 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-q9bwm\"" Jul 06 23:57:20.936919 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.936906 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 23:57:20.965663 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.965607 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6"] Jul 06 23:57:20.966576 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.966555 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-xxxjj"] Jul 06 23:57:20.974626 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:20.974604 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8"] Jul 06 23:57:21.032968 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.032932 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xxxjj\" (UID: \"67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xxxjj" Jul 06 23:57:21.033134 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.033078 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e21bc859-982a-47db-8088-70c4f7a23331-klusterlet-config\") pod \"klusterlet-addon-workmgr-848b8d6595-mwqh8\" (UID: \"e21bc859-982a-47db-8088-70c4f7a23331\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" Jul 06 23:57:21.033134 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.033125 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-xxxjj\" (UID: \"67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xxxjj" Jul 06 23:57:21.033249 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.033198 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e21bc859-982a-47db-8088-70c4f7a23331-tmp\") pod \"klusterlet-addon-workmgr-848b8d6595-mwqh8\" (UID: \"e21bc859-982a-47db-8088-70c4f7a23331\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" Jul 06 23:57:21.033249 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.033241 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf757\" (UniqueName: \"kubernetes.io/projected/e21bc859-982a-47db-8088-70c4f7a23331-kube-api-access-lf757\") pod \"klusterlet-addon-workmgr-848b8d6595-mwqh8\" (UID: \"e21bc859-982a-47db-8088-70c4f7a23331\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" Jul 06 23:57:21.033366 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.033267 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0eea11a-a741-4e5a-be48-ad687b4e7caf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6\" (UID: \"a0eea11a-a741-4e5a-be48-ad687b4e7caf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6" Jul 06 23:57:21.033366 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.033295 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncc9r\" (UniqueName: \"kubernetes.io/projected/a0eea11a-a741-4e5a-be48-ad687b4e7caf-kube-api-access-ncc9r\") pod \"managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6\" (UID: \"a0eea11a-a741-4e5a-be48-ad687b4e7caf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6" Jul 06 23:57:21.050952 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.050922 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d47887bfc-2r99h"] Jul 06 23:57:21.134685 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.134651 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0eea11a-a741-4e5a-be48-ad687b4e7caf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6\" (UID: \"a0eea11a-a741-4e5a-be48-ad687b4e7caf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6" Jul 06 23:57:21.134685 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.134684 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncc9r\" (UniqueName: \"kubernetes.io/projected/a0eea11a-a741-4e5a-be48-ad687b4e7caf-kube-api-access-ncc9r\") pod \"managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6\" (UID: \"a0eea11a-a741-4e5a-be48-ad687b4e7caf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6" Jul 06 23:57:21.134944 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.134741 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xxxjj\" (UID: \"67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xxxjj" Jul 06 23:57:21.134944 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.134758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e21bc859-982a-47db-8088-70c4f7a23331-klusterlet-config\") pod \"klusterlet-addon-workmgr-848b8d6595-mwqh8\" (UID: \"e21bc859-982a-47db-8088-70c4f7a23331\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" Jul 06 23:57:21.134944 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.134778 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-xxxjj\" (UID: \"67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xxxjj" Jul 06 23:57:21.134944 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.134806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e21bc859-982a-47db-8088-70c4f7a23331-tmp\") pod \"klusterlet-addon-workmgr-848b8d6595-mwqh8\" (UID: \"e21bc859-982a-47db-8088-70c4f7a23331\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" Jul 06 23:57:21.134944 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.134826 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf757\" (UniqueName: \"kubernetes.io/projected/e21bc859-982a-47db-8088-70c4f7a23331-kube-api-access-lf757\") pod \"klusterlet-addon-workmgr-848b8d6595-mwqh8\" (UID: \"e21bc859-982a-47db-8088-70c4f7a23331\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" Jul 06 23:57:21.135344 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.135315 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e21bc859-982a-47db-8088-70c4f7a23331-tmp\") pod \"klusterlet-addon-workmgr-848b8d6595-mwqh8\" (UID: \"e21bc859-982a-47db-8088-70c4f7a23331\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" Jul 06 23:57:21.135596 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.135576 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-xxxjj\" (UID: \"67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xxxjj" Jul 06 23:57:21.137261 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.137233 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xxxjj\" (UID: \"67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xxxjj" Jul 06 23:57:21.137693 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.137678 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0eea11a-a741-4e5a-be48-ad687b4e7caf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6\" (UID: \"a0eea11a-a741-4e5a-be48-ad687b4e7caf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6" Jul 06 23:57:21.137826 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.137807 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e21bc859-982a-47db-8088-70c4f7a23331-klusterlet-config\") pod \"klusterlet-addon-workmgr-848b8d6595-mwqh8\" (UID: \"e21bc859-982a-47db-8088-70c4f7a23331\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" Jul 06 23:57:21.140704 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.140680 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dd6b96ffb-hq86m"] Jul 06 23:57:21.143428 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.143402 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.151637 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.151612 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncc9r\" (UniqueName: \"kubernetes.io/projected/a0eea11a-a741-4e5a-be48-ad687b4e7caf-kube-api-access-ncc9r\") pod \"managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6\" (UID: \"a0eea11a-a741-4e5a-be48-ad687b4e7caf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6" Jul 06 23:57:21.156414 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.156391 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf757\" (UniqueName: \"kubernetes.io/projected/e21bc859-982a-47db-8088-70c4f7a23331-kube-api-access-lf757\") pod \"klusterlet-addon-workmgr-848b8d6595-mwqh8\" (UID: \"e21bc859-982a-47db-8088-70c4f7a23331\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" Jul 06 23:57:21.209754 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.209724 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dd6b96ffb-hq86m"] Jul 06 23:57:21.235624 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.235590 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73c6558c-cd86-406f-940a-fae3c81fa41a-bound-sa-token\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.235624 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.235623 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73c6558c-cd86-406f-940a-fae3c81fa41a-registry-certificates\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.235850 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.235642 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73c6558c-cd86-406f-940a-fae3c81fa41a-registry-tls\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.235850 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.235689 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73c6558c-cd86-406f-940a-fae3c81fa41a-trusted-ca\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.235850 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.235782 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb5sq\" (UniqueName: \"kubernetes.io/projected/73c6558c-cd86-406f-940a-fae3c81fa41a-kube-api-access-rb5sq\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.235850 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.235827 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73c6558c-cd86-406f-940a-fae3c81fa41a-ca-trust-extracted\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.235989 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.235887 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73c6558c-cd86-406f-940a-fae3c81fa41a-image-registry-private-configuration\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.235989 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.235913 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73c6558c-cd86-406f-940a-fae3c81fa41a-installation-pull-secrets\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.249714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.249684 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6" Jul 06 23:57:21.256623 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.256603 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" Jul 06 23:57:21.262930 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.262897 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xxxjj" Jul 06 23:57:21.338385 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.336882 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73c6558c-cd86-406f-940a-fae3c81fa41a-registry-certificates\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.338385 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.336927 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73c6558c-cd86-406f-940a-fae3c81fa41a-registry-tls\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.338385 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.336966 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73c6558c-cd86-406f-940a-fae3c81fa41a-trusted-ca\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.338385 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.337010 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb5sq\" (UniqueName: \"kubernetes.io/projected/73c6558c-cd86-406f-940a-fae3c81fa41a-kube-api-access-rb5sq\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.338385 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.337048 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73c6558c-cd86-406f-940a-fae3c81fa41a-ca-trust-extracted\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.338385 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.337111 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73c6558c-cd86-406f-940a-fae3c81fa41a-image-registry-private-configuration\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.338385 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.337148 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73c6558c-cd86-406f-940a-fae3c81fa41a-installation-pull-secrets\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.338385 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.337209 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73c6558c-cd86-406f-940a-fae3c81fa41a-bound-sa-token\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.338385 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.337974 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73c6558c-cd86-406f-940a-fae3c81fa41a-ca-trust-extracted\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.338949 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.338453 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73c6558c-cd86-406f-940a-fae3c81fa41a-registry-certificates\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.339941 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.339917 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73c6558c-cd86-406f-940a-fae3c81fa41a-trusted-ca\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.341231 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.341203 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73c6558c-cd86-406f-940a-fae3c81fa41a-image-registry-private-configuration\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.341634 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.341618 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73c6558c-cd86-406f-940a-fae3c81fa41a-installation-pull-secrets\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.342246 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.342226 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73c6558c-cd86-406f-940a-fae3c81fa41a-registry-tls\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.354786 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.354749 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73c6558c-cd86-406f-940a-fae3c81fa41a-bound-sa-token\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.357772 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.357750 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb5sq\" (UniqueName: \"kubernetes.io/projected/73c6558c-cd86-406f-940a-fae3c81fa41a-kube-api-access-rb5sq\") pod \"image-registry-5dd6b96ffb-hq86m\" (UID: \"73c6558c-cd86-406f-940a-fae3c81fa41a\") " pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.409312 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.409284 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6"] Jul 06 23:57:21.412197 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:57:21.412174 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0eea11a_a741_4e5a_be48_ad687b4e7caf.slice/crio-a69dd7c8590067fd3d8e45195b314a46c44ee19e56063c50bf45bd91e44bc56d WatchSource:0}: Error finding container a69dd7c8590067fd3d8e45195b314a46c44ee19e56063c50bf45bd91e44bc56d: Status 404 returned error can't find the container with id a69dd7c8590067fd3d8e45195b314a46c44ee19e56063c50bf45bd91e44bc56d Jul 06 23:57:21.464840 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.464757 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.607262 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.607235 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dd6b96ffb-hq86m"] Jul 06 23:57:21.613223 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:57:21.613198 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c6558c_cd86_406f_940a_fae3c81fa41a.slice/crio-baa8333b69824178b562bc23a94c4ba2c1238da5042963d7c7e1f0225b6d46b2 WatchSource:0}: Error finding container baa8333b69824178b562bc23a94c4ba2c1238da5042963d7c7e1f0225b6d46b2: Status 404 returned error can't find the container with id baa8333b69824178b562bc23a94c4ba2c1238da5042963d7c7e1f0225b6d46b2 Jul 06 23:57:21.633996 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.633970 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-xxxjj"] Jul 06 23:57:21.637640 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:57:21.637617 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bc5dc2_928f_4a3c_b3df_5b0f0fb1d7ee.slice/crio-38a7010f6fbf98e864e95def2d03f54dcc9aa999fbc2b4c5324bc315753bc90f WatchSource:0}: Error finding container 38a7010f6fbf98e864e95def2d03f54dcc9aa999fbc2b4c5324bc315753bc90f: Status 404 returned error can't find the container with id 38a7010f6fbf98e864e95def2d03f54dcc9aa999fbc2b4c5324bc315753bc90f Jul 06 23:57:21.639887 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.639865 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8"] Jul 06 23:57:21.642882 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:57:21.642859 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21bc859_982a_47db_8088_70c4f7a23331.slice/crio-bababf4053f0993043468584fccdd5c0c27356c2f2ee4731a103136cda1b6acb WatchSource:0}: Error finding container bababf4053f0993043468584fccdd5c0c27356c2f2ee4731a103136cda1b6acb: Status 404 returned error can't find the container with id bababf4053f0993043468584fccdd5c0c27356c2f2ee4731a103136cda1b6acb Jul 06 23:57:21.940168 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.939715 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" event={"ID":"73c6558c-cd86-406f-940a-fae3c81fa41a","Type":"ContainerStarted","Data":"5f02380dff1070d309e2a2560bce5b870f564e20b29cbe51df82f0576f66c6f3"} Jul 06 23:57:21.940168 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.939768 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" event={"ID":"73c6558c-cd86-406f-940a-fae3c81fa41a","Type":"ContainerStarted","Data":"baa8333b69824178b562bc23a94c4ba2c1238da5042963d7c7e1f0225b6d46b2"} Jul 06 23:57:21.940168 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.939810 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:21.941089 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.941062 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xxxjj" event={"ID":"67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee","Type":"ContainerStarted","Data":"38a7010f6fbf98e864e95def2d03f54dcc9aa999fbc2b4c5324bc315753bc90f"} Jul 06 23:57:21.943649 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.943593 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6" event={"ID":"a0eea11a-a741-4e5a-be48-ad687b4e7caf","Type":"ContainerStarted","Data":"a69dd7c8590067fd3d8e45195b314a46c44ee19e56063c50bf45bd91e44bc56d"} Jul 06 23:57:21.946633 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.946565 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" event={"ID":"e21bc859-982a-47db-8088-70c4f7a23331","Type":"ContainerStarted","Data":"bababf4053f0993043468584fccdd5c0c27356c2f2ee4731a103136cda1b6acb"} Jul 06 23:57:21.966547 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:21.966489 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" podStartSLOduration=0.966468085 podStartE2EDuration="966.468085ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:21.965403492 +0000 UTC m=+65.105860574" watchObservedRunningTime="2026-07-06 23:57:21.966468085 +0000 UTC m=+65.106925143" Jul 06 23:57:22.906528 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:22.906491 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4txr6" Jul 06 23:57:23.153123 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.153080 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:57:23.155708 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.155685 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 23:57:23.166492 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.166405 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76bea87f-a057-4a59-8e4a-171542fa018a-original-pull-secret\") pod \"global-pull-secret-syncer-x5k59\" (UID: \"76bea87f-a057-4a59-8e4a-171542fa018a\") " pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:57:23.254193 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.254153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55bq\" (UniqueName: \"kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq\") pod \"network-check-target-bcn92\" (UID: \"388db162-a514-4da6-b4ad-398a94e22e01\") " pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:57:23.254377 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.254214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs\") pod \"network-metrics-daemon-mddqt\" (UID: \"b8d4e21f-d7f0-404d-a8e8-4da642d240c7\") " pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:57:23.257382 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.257347 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 23:57:23.257557 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.257527 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 23:57:23.267742 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.267714 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 23:57:23.267893 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.267873 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d4e21f-d7f0-404d-a8e8-4da642d240c7-metrics-certs\") pod \"network-metrics-daemon-mddqt\" (UID: \"b8d4e21f-d7f0-404d-a8e8-4da642d240c7\") " pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:57:23.278751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.278729 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k55bq\" (UniqueName: \"kubernetes.io/projected/388db162-a514-4da6-b4ad-398a94e22e01-kube-api-access-k55bq\") pod \"network-check-target-bcn92\" (UID: \"388db162-a514-4da6-b4ad-398a94e22e01\") " pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:57:23.394132 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.394091 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5k59" Jul 06 23:57:23.404203 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.404176 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-knf8z\"" Jul 06 23:57:23.411957 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.411937 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mddqt" Jul 06 23:57:23.412470 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.412450 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lswch\"" Jul 06 23:57:23.418314 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:23.418253 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:57:24.486459 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:24.486366 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dntg6"] Jul 06 23:57:24.489568 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:24.489545 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dntg6" Jul 06 23:57:24.492062 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:24.492019 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9ctzd\"" Jul 06 23:57:24.492266 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:24.492250 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 06 23:57:24.503048 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:24.503026 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dntg6"] Jul 06 23:57:24.565535 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:24.565504 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ca82b883-02f0-4aeb-a8b9-4b0a07b4069f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-dntg6\" (UID: \"ca82b883-02f0-4aeb-a8b9-4b0a07b4069f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dntg6" Jul 06 23:57:24.666785 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:24.666752 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ca82b883-02f0-4aeb-a8b9-4b0a07b4069f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-dntg6\" (UID: \"ca82b883-02f0-4aeb-a8b9-4b0a07b4069f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dntg6" Jul 06 23:57:24.669443 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:24.669401 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ca82b883-02f0-4aeb-a8b9-4b0a07b4069f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-dntg6\" (UID: \"ca82b883-02f0-4aeb-a8b9-4b0a07b4069f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dntg6" Jul 06 23:57:24.799579 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:24.799489 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dntg6" Jul 06 23:57:25.642711 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:25.642684 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bcn92"] Jul 06 23:57:25.645140 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:57:25.645106 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388db162_a514_4da6_b4ad_398a94e22e01.slice/crio-91e91762b5c47e9d081d51e2c3a4642bb11647e820f0decff04668a5254cc755 WatchSource:0}: Error finding container 91e91762b5c47e9d081d51e2c3a4642bb11647e820f0decff04668a5254cc755: Status 404 returned error can't find the container with id 91e91762b5c47e9d081d51e2c3a4642bb11647e820f0decff04668a5254cc755 Jul 06 23:57:25.859499 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:25.859468 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dntg6"] Jul 06 23:57:25.862288 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:25.862266 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x5k59"] Jul 06 23:57:25.862729 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:57:25.862708 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca82b883_02f0_4aeb_a8b9_4b0a07b4069f.slice/crio-38ed29295a9ecf6edda2a4d865c7e4f8f98cb0f4b93f40fa220f810aacb860e4 WatchSource:0}: Error finding container 38ed29295a9ecf6edda2a4d865c7e4f8f98cb0f4b93f40fa220f810aacb860e4: Status 404 returned error can't find the container with id 38ed29295a9ecf6edda2a4d865c7e4f8f98cb0f4b93f40fa220f810aacb860e4 Jul 06 23:57:25.865833 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:25.865813 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mddqt"] Jul 06 23:57:25.867589 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:57:25.867557 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bea87f_a057_4a59_8e4a_171542fa018a.slice/crio-0269cfc6e5307a8bf96775624e790b4f2514919b4d010700f90c771d35c346fc WatchSource:0}: Error finding container 0269cfc6e5307a8bf96775624e790b4f2514919b4d010700f90c771d35c346fc: Status 404 returned error can't find the container with id 0269cfc6e5307a8bf96775624e790b4f2514919b4d010700f90c771d35c346fc Jul 06 23:57:25.869396 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:57:25.869379 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d4e21f_d7f0_404d_a8e8_4da642d240c7.slice/crio-6e5af4986279136c478ca67bcac92bc4f111241f833127ca4ac886d1639dd913 WatchSource:0}: Error finding container 6e5af4986279136c478ca67bcac92bc4f111241f833127ca4ac886d1639dd913: Status 404 returned error can't find the container with id 6e5af4986279136c478ca67bcac92bc4f111241f833127ca4ac886d1639dd913 Jul 06 23:57:25.968121 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:25.968083 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xxxjj" event={"ID":"67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee","Type":"ContainerStarted","Data":"ad41ba1b9daf722351d275620d5bb000098722d6049e58b77b8d8dbf6991fd3e"} Jul 06 23:57:25.969287 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:25.969261 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mddqt" event={"ID":"b8d4e21f-d7f0-404d-a8e8-4da642d240c7","Type":"ContainerStarted","Data":"6e5af4986279136c478ca67bcac92bc4f111241f833127ca4ac886d1639dd913"} Jul 06 23:57:25.970666 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:25.970634 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6" event={"ID":"a0eea11a-a741-4e5a-be48-ad687b4e7caf","Type":"ContainerStarted","Data":"c20418ed9beb766e6da2163fe6276f616e5812ab1689566a236ae61aaf719afd"} Jul 06 23:57:25.972281 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:25.972260 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" event={"ID":"e21bc859-982a-47db-8088-70c4f7a23331","Type":"ContainerStarted","Data":"ad852dd25d78981553c29192672de7bd50b2bbb318b3450ac899e61f6f073097"} Jul 06 23:57:25.972377 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:25.972363 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" Jul 06 23:57:25.973538 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:25.973517 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dntg6" event={"ID":"ca82b883-02f0-4aeb-a8b9-4b0a07b4069f","Type":"ContainerStarted","Data":"38ed29295a9ecf6edda2a4d865c7e4f8f98cb0f4b93f40fa220f810aacb860e4"} Jul 06 23:57:25.974234 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:25.974216 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" Jul 06 23:57:25.974755 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:25.974720 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bcn92" event={"ID":"388db162-a514-4da6-b4ad-398a94e22e01","Type":"ContainerStarted","Data":"91e91762b5c47e9d081d51e2c3a4642bb11647e820f0decff04668a5254cc755"} Jul 06 23:57:25.975861 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:25.975844 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x5k59" event={"ID":"76bea87f-a057-4a59-8e4a-171542fa018a","Type":"ContainerStarted","Data":"0269cfc6e5307a8bf96775624e790b4f2514919b4d010700f90c771d35c346fc"} Jul 06 23:57:25.985794 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:25.985747 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xxxjj" podStartSLOduration=2.204271132 podStartE2EDuration="5.985733704s" totalImagesPullingTime="3.781462572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:25.984656812 +0000 UTC m=+69.125113867" watchObservedRunningTime="2026-07-06 23:57:25.985733704 +0000 UTC m=+69.126190754" Jul 06 23:57:26.008973 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:26.008929 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9f4d4d8-cs8s6" podStartSLOduration=1.993560556 podStartE2EDuration="6.008914303s" totalImagesPullingTime="4.015353747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:26.008132623 +0000 UTC m=+69.148589679" watchObservedRunningTime="2026-07-06 23:57:26.008914303 +0000 UTC m=+69.149371358" Jul 06 23:57:26.026905 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:26.026864 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848b8d6595-mwqh8" podStartSLOduration=2.218702459 podStartE2EDuration="6.026850706s" totalImagesPullingTime="3.808148247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:26.025943032 +0000 UTC m=+69.166400089" watchObservedRunningTime="2026-07-06 23:57:26.026850706 +0000 UTC m=+69.167307762" Jul 06 23:57:27.986407 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:27.986273 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dntg6" event={"ID":"ca82b883-02f0-4aeb-a8b9-4b0a07b4069f","Type":"ContainerStarted","Data":"50869bfaa97cdec7d00f7aa732834e15cd551b4b51df615fda0c3bdbcf86a336"} Jul 06 23:57:27.986904 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:27.986612 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dntg6" Jul 06 23:57:27.988504 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:27.988464 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mddqt" event={"ID":"b8d4e21f-d7f0-404d-a8e8-4da642d240c7","Type":"ContainerStarted","Data":"33b64c11f4a1f02c5da1d9e6fecfdb12c69f5a87bd64a33744c287d33157b7c1"} Jul 06 23:57:27.992625 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:27.992602 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dntg6" Jul 06 23:57:28.005095 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:28.004767 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dntg6" podStartSLOduration=2.139507302 podStartE2EDuration="4.00475044s" totalImagesPullingTime="1.865243138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:28.002567338 +0000 UTC m=+71.143024398" watchObservedRunningTime="2026-07-06 23:57:28.00475044 +0000 UTC m=+71.145207496" Jul 06 23:57:31.001114 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:31.001081 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mddqt" event={"ID":"b8d4e21f-d7f0-404d-a8e8-4da642d240c7","Type":"ContainerStarted","Data":"2b712a05b474a51e1ec8315769397ce8378dbd8107cf2c7028668af6e209c3cf"} Jul 06 23:57:31.002547 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:31.002504 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bcn92" event={"ID":"388db162-a514-4da6-b4ad-398a94e22e01","Type":"ContainerStarted","Data":"d9300ea837293872a916d205e5ef142204f33cfc604d1d98fc8cf9192c280dcb"} Jul 06 23:57:31.002709 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:31.002637 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:57:31.003776 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:31.003755 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x5k59" event={"ID":"76bea87f-a057-4a59-8e4a-171542fa018a","Type":"ContainerStarted","Data":"b0a7795924f81f6357565422c8036e6ec9b50ad623c8b03b75e75bdf399143e6"} Jul 06 23:57:31.020731 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:31.020691 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mddqt" podStartSLOduration=71.163993123 podStartE2EDuration="1m13.020679572s" totalImagesPullingTime="1.856686449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:31.019795954 +0000 UTC m=+74.160253008" watchObservedRunningTime="2026-07-06 23:57:31.020679572 +0000 UTC m=+74.161136628" Jul 06 23:57:31.050834 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:31.050798 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bcn92" podStartSLOduration=68.276218175 podStartE2EDuration="1m13.050786334s" totalImagesPullingTime="4.774568159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:31.035162763 +0000 UTC m=+74.175619819" watchObservedRunningTime="2026-07-06 23:57:31.050786334 +0000 UTC m=+74.191243389" Jul 06 23:57:31.051000 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:31.050983 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x5k59" podStartSLOduration=68.495743351 podStartE2EDuration="1m13.050978342s" totalImagesPullingTime="4.555234991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:31.049927461 +0000 UTC m=+74.190384518" watchObservedRunningTime="2026-07-06 23:57:31.050978342 +0000 UTC m=+74.191435398" Jul 06 23:57:31.055777 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:31.055754 2568 patch_prober.go:28] interesting pod/image-registry-5d47887bfc-2r99h container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 23:57:31.055888 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:31.055795 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" podUID="e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 23:57:37.035692 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.035661 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xsr4v"] Jul 06 23:57:37.041151 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.041130 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.043704 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.043683 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 23:57:37.043887 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.043869 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 23:57:37.044113 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.043896 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 23:57:37.044182 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.043788 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f7qp4\"" Jul 06 23:57:37.044531 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.044516 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 23:57:37.044757 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.044734 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 23:57:37.044930 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.044895 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 23:57:37.164050 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.164016 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-wtmp\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.164210 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.164065 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcm5b\" (UniqueName: \"kubernetes.io/projected/ccb34dac-83d6-4197-88ce-b709d828ff52-kube-api-access-wcm5b\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.164210 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.164128 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ccb34dac-83d6-4197-88ce-b709d828ff52-root\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.164210 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.164156 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-tls\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.164210 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.164179 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccb34dac-83d6-4197-88ce-b709d828ff52-sys\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.164342 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.164215 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-textfile\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.164342 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.164259 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb34dac-83d6-4197-88ce-b709d828ff52-metrics-client-ca\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.164342 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.164294 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-accelerators-collector-config\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.164342 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.164329 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.265781 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.265727 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.265781 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.265774 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-wtmp\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.266017 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.265811 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcm5b\" (UniqueName: \"kubernetes.io/projected/ccb34dac-83d6-4197-88ce-b709d828ff52-kube-api-access-wcm5b\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.266017 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.265847 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ccb34dac-83d6-4197-88ce-b709d828ff52-root\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.266017 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.265874 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-tls\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.266017 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.265897 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccb34dac-83d6-4197-88ce-b709d828ff52-sys\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.266017 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.265941 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-textfile\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.266017 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.265975 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb34dac-83d6-4197-88ce-b709d828ff52-metrics-client-ca\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.266297 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.266017 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-accelerators-collector-config\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.266847 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.266414 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ccb34dac-83d6-4197-88ce-b709d828ff52-root\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.266847 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.266584 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccb34dac-83d6-4197-88ce-b709d828ff52-sys\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.266847 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.266660 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-accelerators-collector-config\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.267044 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.266914 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-textfile\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.267344 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.267321 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-wtmp\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.267471 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.267378 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb34dac-83d6-4197-88ce-b709d828ff52-metrics-client-ca\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.269913 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.269870 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-tls\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.271052 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.271011 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccb34dac-83d6-4197-88ce-b709d828ff52-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.276076 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.276052 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcm5b\" (UniqueName: \"kubernetes.io/projected/ccb34dac-83d6-4197-88ce-b709d828ff52-kube-api-access-wcm5b\") pod \"node-exporter-xsr4v\" (UID: \"ccb34dac-83d6-4197-88ce-b709d828ff52\") " pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.352068 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:37.351989 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xsr4v" Jul 06 23:57:37.363373 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:57:37.363332 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb34dac_83d6_4197_88ce_b709d828ff52.slice/crio-b4085e9ca7c562b080d646b7895409a15ef3979ec1c33d9a901eb344c70e3cde WatchSource:0}: Error finding container b4085e9ca7c562b080d646b7895409a15ef3979ec1c33d9a901eb344c70e3cde: Status 404 returned error can't find the container with id b4085e9ca7c562b080d646b7895409a15ef3979ec1c33d9a901eb344c70e3cde Jul 06 23:57:38.030021 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:38.029975 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xsr4v" event={"ID":"ccb34dac-83d6-4197-88ce-b709d828ff52","Type":"ContainerStarted","Data":"b4085e9ca7c562b080d646b7895409a15ef3979ec1c33d9a901eb344c70e3cde"} Jul 06 23:57:39.035920 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:39.035792 2568 generic.go:356] "Generic (PLEG): container finished" podID="ccb34dac-83d6-4197-88ce-b709d828ff52" containerID="371ea2360556df3689026525afc1b0847298fcb3211abf99566952e3fdc358fc" exitCode=0 Jul 06 23:57:39.035920 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:39.035887 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xsr4v" event={"ID":"ccb34dac-83d6-4197-88ce-b709d828ff52","Type":"ContainerDied","Data":"371ea2360556df3689026525afc1b0847298fcb3211abf99566952e3fdc358fc"} Jul 06 23:57:40.043496 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:40.043461 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xsr4v" event={"ID":"ccb34dac-83d6-4197-88ce-b709d828ff52","Type":"ContainerStarted","Data":"ba3cdcb89c6a5ab819681fa4abfbe170cae5e1dbd72a39c565ce9cc7c683aa9e"} Jul 06 23:57:40.043891 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:40.043519 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xsr4v" event={"ID":"ccb34dac-83d6-4197-88ce-b709d828ff52","Type":"ContainerStarted","Data":"91a6adb4d9b24791eb620aaebdf0faa88797a306ff2eb04b9c15878d40baa5f5"} Jul 06 23:57:40.066928 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:40.066866 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xsr4v" podStartSLOduration=2.017983924 podStartE2EDuration="3.066845448s" totalImagesPullingTime="984.958055ms" totalInitContainerRuntime="63.903469ms" isStatefulPod=true podCreationTimestamp="2026-07-06 23:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:40.06519747 +0000 UTC m=+83.205654530" watchObservedRunningTime="2026-07-06 23:57:40.066845448 +0000 UTC m=+83.207302505" Jul 06 23:57:41.055400 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:41.055373 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:57:42.956481 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:42.956455 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5dd6b96ffb-hq86m" Jul 06 23:57:46.070637 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:46.070565 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" podUID="e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" containerName="registry" containerID="cri-o://d604bb738a467e96ba439523614ef3cdab3d8224295d5fb008d9c3bef455b513" gracePeriod=30 Jul 06 23:57:46.886259 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:46.886234 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:57:47.042132 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.042049 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d482c\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-kube-api-access-d482c\") pod \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " Jul 06 23:57:47.042132 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.042108 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls\") pod \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " Jul 06 23:57:47.042132 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.042131 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-bound-sa-token\") pod \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " Jul 06 23:57:47.042482 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.042161 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-ca-trust-extracted\") pod \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " Jul 06 23:57:47.042482 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.042187 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-trusted-ca\") pod \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " Jul 06 23:57:47.042482 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.042205 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-image-registry-private-configuration\") pod \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " Jul 06 23:57:47.042482 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.042229 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-certificates\") pod \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " Jul 06 23:57:47.042482 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.042395 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-installation-pull-secrets\") pod \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\" (UID: \"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c\") " Jul 06 23:57:47.042745 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.042648 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" (UID: "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:57:47.042959 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.042927 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" (UID: "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:57:47.044736 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.044714 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" (UID: "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:57:47.044833 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.044786 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" (UID: "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:57:47.044833 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.044826 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" (UID: "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:57:47.045010 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.044982 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" (UID: "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:57:47.045047 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.045007 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-kube-api-access-d482c" (OuterVolumeSpecName: "kube-api-access-d482c") pod "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" (UID: "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c"). InnerVolumeSpecName "kube-api-access-d482c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:57:47.050832 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.050810 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" (UID: "e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:57:47.072026 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.072000 2568 generic.go:356] "Generic (PLEG): container finished" podID="e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" containerID="d604bb738a467e96ba439523614ef3cdab3d8224295d5fb008d9c3bef455b513" exitCode=0 Jul 06 23:57:47.072326 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.072036 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" Jul 06 23:57:47.072326 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.072039 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" event={"ID":"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c","Type":"ContainerDied","Data":"d604bb738a467e96ba439523614ef3cdab3d8224295d5fb008d9c3bef455b513"} Jul 06 23:57:47.072326 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.072080 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d47887bfc-2r99h" event={"ID":"e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c","Type":"ContainerDied","Data":"41c8f19a4c51457d35e720b48a33d04e7afe334d79ee52829e622172d8a4df7b"} Jul 06 23:57:47.072326 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.072101 2568 scope.go:117] "RemoveContainer" containerID="d604bb738a467e96ba439523614ef3cdab3d8224295d5fb008d9c3bef455b513" Jul 06 23:57:47.080249 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.080234 2568 scope.go:117] "RemoveContainer" containerID="d604bb738a467e96ba439523614ef3cdab3d8224295d5fb008d9c3bef455b513" Jul 06 23:57:47.080520 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:57:47.080502 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d604bb738a467e96ba439523614ef3cdab3d8224295d5fb008d9c3bef455b513\": container with ID starting with d604bb738a467e96ba439523614ef3cdab3d8224295d5fb008d9c3bef455b513 not found: ID does not exist" containerID="d604bb738a467e96ba439523614ef3cdab3d8224295d5fb008d9c3bef455b513" Jul 06 23:57:47.080573 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.080530 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d604bb738a467e96ba439523614ef3cdab3d8224295d5fb008d9c3bef455b513"} err="failed to get container status \"d604bb738a467e96ba439523614ef3cdab3d8224295d5fb008d9c3bef455b513\": rpc error: code = NotFound desc = could not find container \"d604bb738a467e96ba439523614ef3cdab3d8224295d5fb008d9c3bef455b513\": container with ID starting with d604bb738a467e96ba439523614ef3cdab3d8224295d5fb008d9c3bef455b513 not found: ID does not exist" Jul 06 23:57:47.091595 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.091576 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d47887bfc-2r99h"] Jul 06 23:57:47.095742 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.095724 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d47887bfc-2r99h"] Jul 06 23:57:47.143699 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.143680 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-bound-sa-token\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:57:47.143699 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.143697 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-ca-trust-extracted\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:57:47.143816 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.143706 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-trusted-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:57:47.143816 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.143717 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-image-registry-private-configuration\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:57:47.143816 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.143725 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-certificates\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:57:47.143816 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.143733 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-installation-pull-secrets\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:57:47.143816 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.143741 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d482c\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-kube-api-access-d482c\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:57:47.143816 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.143748 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c-registry-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:57:47.488688 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.488660 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" path="/var/lib/kubelet/pods/e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c/volumes" Jul 06 23:57:47.659134 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.659102 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-76gvs"] Jul 06 23:57:47.659353 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.659342 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" containerName="registry" Jul 06 23:57:47.659390 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.659354 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" containerName="registry" Jul 06 23:57:47.659465 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.659408 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7976dfa-a4b1-46b6-b0f1-ac2a7e95454c" containerName="registry" Jul 06 23:57:47.662450 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.662436 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-76gvs" Jul 06 23:57:47.665231 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.665211 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-l6klp\"" Jul 06 23:57:47.665339 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.665218 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 23:57:47.665470 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.665455 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 23:57:47.671675 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.671657 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-76gvs"] Jul 06 23:57:47.747044 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.746981 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n989c\" (UniqueName: \"kubernetes.io/projected/6f4ba1c0-2197-47b3-872c-564ecba6c44e-kube-api-access-n989c\") pod \"downloads-685bfb5f64-76gvs\" (UID: \"6f4ba1c0-2197-47b3-872c-564ecba6c44e\") " pod="openshift-console/downloads-685bfb5f64-76gvs" Jul 06 23:57:47.848061 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.848035 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n989c\" (UniqueName: \"kubernetes.io/projected/6f4ba1c0-2197-47b3-872c-564ecba6c44e-kube-api-access-n989c\") pod \"downloads-685bfb5f64-76gvs\" (UID: \"6f4ba1c0-2197-47b3-872c-564ecba6c44e\") " pod="openshift-console/downloads-685bfb5f64-76gvs" Jul 06 23:57:47.857047 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.857016 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n989c\" (UniqueName: \"kubernetes.io/projected/6f4ba1c0-2197-47b3-872c-564ecba6c44e-kube-api-access-n989c\") pod \"downloads-685bfb5f64-76gvs\" (UID: \"6f4ba1c0-2197-47b3-872c-564ecba6c44e\") " pod="openshift-console/downloads-685bfb5f64-76gvs" Jul 06 23:57:47.971005 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:47.970966 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-76gvs" Jul 06 23:57:48.097474 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:48.097444 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-76gvs"] Jul 06 23:57:48.100993 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:57:48.100957 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4ba1c0_2197_47b3_872c_564ecba6c44e.slice/crio-4d0f1c212ad79b79ec7a3a9d5ebfc9da87ece07f67693df969ba2dcd3ee10aea WatchSource:0}: Error finding container 4d0f1c212ad79b79ec7a3a9d5ebfc9da87ece07f67693df969ba2dcd3ee10aea: Status 404 returned error can't find the container with id 4d0f1c212ad79b79ec7a3a9d5ebfc9da87ece07f67693df969ba2dcd3ee10aea Jul 06 23:57:49.082479 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:49.082440 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-76gvs" event={"ID":"6f4ba1c0-2197-47b3-872c-564ecba6c44e","Type":"ContainerStarted","Data":"4d0f1c212ad79b79ec7a3a9d5ebfc9da87ece07f67693df969ba2dcd3ee10aea"} Jul 06 23:57:56.795860 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:56.795826 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75c7867648-cxrs4"] Jul 06 23:57:56.900271 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:56.900229 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c7867648-cxrs4"] Jul 06 23:57:56.900483 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:56.900353 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:56.902768 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:56.902748 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 23:57:56.903024 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:56.903004 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 23:57:56.903024 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:56.903017 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 23:57:56.904236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:56.904016 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 23:57:56.904236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:56.904054 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pdnjh\"" Jul 06 23:57:56.904236 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:56.904086 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 23:57:57.021901 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.021869 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-oauth-serving-cert\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.022059 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.021918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-service-ca\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.022059 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.022000 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-console-config\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.022152 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.022070 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn648\" (UniqueName: \"kubernetes.io/projected/d428aa6d-8373-4d21-b00e-96a38bef4716-kube-api-access-kn648\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.022152 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.022125 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d428aa6d-8373-4d21-b00e-96a38bef4716-console-serving-cert\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.022224 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.022151 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d428aa6d-8373-4d21-b00e-96a38bef4716-console-oauth-config\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.123574 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.123383 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-service-ca\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.123574 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.123498 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-console-config\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.123574 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.123539 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn648\" (UniqueName: \"kubernetes.io/projected/d428aa6d-8373-4d21-b00e-96a38bef4716-kube-api-access-kn648\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.123800 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.123589 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d428aa6d-8373-4d21-b00e-96a38bef4716-console-serving-cert\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.123800 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.123616 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d428aa6d-8373-4d21-b00e-96a38bef4716-console-oauth-config\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.123800 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.123660 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-oauth-serving-cert\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.124198 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.124169 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-service-ca\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.124325 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.124247 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-console-config\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.124325 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.124266 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-oauth-serving-cert\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.126155 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.126108 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d428aa6d-8373-4d21-b00e-96a38bef4716-console-serving-cert\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.126499 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.126477 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d428aa6d-8373-4d21-b00e-96a38bef4716-console-oauth-config\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.136666 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.136644 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn648\" (UniqueName: \"kubernetes.io/projected/d428aa6d-8373-4d21-b00e-96a38bef4716-kube-api-access-kn648\") pod \"console-75c7867648-cxrs4\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:57:57.211124 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:57:57.211098 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:58:02.010846 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:02.010815 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bcn92" Jul 06 23:58:04.142874 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:04.142845 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c7867648-cxrs4"] Jul 06 23:58:04.146263 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:58:04.146234 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd428aa6d_8373_4d21_b00e_96a38bef4716.slice/crio-33bb0eb059d1851203d89a7564d2f8459759bbe94b46d9f82764cc414b664255 WatchSource:0}: Error finding container 33bb0eb059d1851203d89a7564d2f8459759bbe94b46d9f82764cc414b664255: Status 404 returned error can't find the container with id 33bb0eb059d1851203d89a7564d2f8459759bbe94b46d9f82764cc414b664255 Jul 06 23:58:05.145500 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.145456 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-76gvs" event={"ID":"6f4ba1c0-2197-47b3-872c-564ecba6c44e","Type":"ContainerStarted","Data":"e11b76b2e7cf3469a67fc8707cf20b3439bdf9b6ff7d0af8d66d35ada775865a"} Jul 06 23:58:05.146703 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.146047 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-76gvs" Jul 06 23:58:05.148041 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.148013 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c7867648-cxrs4" event={"ID":"d428aa6d-8373-4d21-b00e-96a38bef4716","Type":"ContainerStarted","Data":"33bb0eb059d1851203d89a7564d2f8459759bbe94b46d9f82764cc414b664255"} Jul 06 23:58:05.164105 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.164021 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-76gvs" Jul 06 23:58:05.165676 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.165629 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-76gvs" podStartSLOduration=1.8510547069999999 podStartE2EDuration="18.165609392s" totalImagesPullingTime="16.314554685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:58:05.165005163 +0000 UTC m=+108.305462243" watchObservedRunningTime="2026-07-06 23:58:05.165609392 +0000 UTC m=+108.306066450" Jul 06 23:58:05.565216 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.565176 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c48c4fdf8-dzzpr"] Jul 06 23:58:05.594197 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.593689 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c48c4fdf8-dzzpr"] Jul 06 23:58:05.594197 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.593808 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.610630 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.610411 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 23:58:05.695651 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.695610 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-oauth-serving-cert\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.695855 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.695659 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px97t\" (UniqueName: \"kubernetes.io/projected/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-kube-api-access-px97t\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.695855 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.695687 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-service-ca\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.695855 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.695713 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-trusted-ca-bundle\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.695855 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.695739 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-serving-cert\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.695855 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.695762 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-oauth-config\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.695855 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.695816 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-config\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.796784 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.796746 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-oauth-serving-cert\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.797009 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.796793 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px97t\" (UniqueName: \"kubernetes.io/projected/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-kube-api-access-px97t\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.797009 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.796814 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-service-ca\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.797009 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.796836 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-trusted-ca-bundle\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.797009 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.796866 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-serving-cert\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.797009 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.796887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-oauth-config\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.797009 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.796931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-config\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.797793 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.797768 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-config\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.798345 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.798314 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-oauth-serving-cert\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.799224 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.799200 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-service-ca\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.800342 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.800319 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-trusted-ca-bundle\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.801677 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.801593 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-oauth-config\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.802579 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.802560 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-serving-cert\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.809861 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.809840 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px97t\" (UniqueName: \"kubernetes.io/projected/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-kube-api-access-px97t\") pod \"console-c48c4fdf8-dzzpr\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:05.907258 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:05.906879 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:06.104600 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:06.104565 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c48c4fdf8-dzzpr"] Jul 06 23:58:06.110429 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:58:06.110382 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38f5aa1_b6d5_4735_8cd0_d36d545185fd.slice/crio-6ad70a91a2062c6542628263a5e1a41d9c6ae111f075cd02d08717fba59cf974 WatchSource:0}: Error finding container 6ad70a91a2062c6542628263a5e1a41d9c6ae111f075cd02d08717fba59cf974: Status 404 returned error can't find the container with id 6ad70a91a2062c6542628263a5e1a41d9c6ae111f075cd02d08717fba59cf974 Jul 06 23:58:06.153999 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:06.153958 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c48c4fdf8-dzzpr" event={"ID":"c38f5aa1-b6d5-4735-8cd0-d36d545185fd","Type":"ContainerStarted","Data":"6ad70a91a2062c6542628263a5e1a41d9c6ae111f075cd02d08717fba59cf974"} Jul 06 23:58:08.172157 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:08.172062 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c7867648-cxrs4" event={"ID":"d428aa6d-8373-4d21-b00e-96a38bef4716","Type":"ContainerStarted","Data":"d13351da8d3723cdb287b2d2609d529cdd413549c6155a9bf261211da8a91e19"} Jul 06 23:58:08.174086 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:08.174051 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c48c4fdf8-dzzpr" event={"ID":"c38f5aa1-b6d5-4735-8cd0-d36d545185fd","Type":"ContainerStarted","Data":"81c2fbf3a4f17311783d77da43eeec39edd8949d9d7a0e777de7185c6ecabe7b"} Jul 06 23:58:08.191618 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:08.191559 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75c7867648-cxrs4" podStartSLOduration=8.369288871 podStartE2EDuration="12.1915414s" totalImagesPullingTime="3.822252529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:58:08.190315308 +0000 UTC m=+111.330772365" watchObservedRunningTime="2026-07-06 23:58:08.1915414 +0000 UTC m=+111.331998456" Jul 06 23:58:08.209008 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:08.207970 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c48c4fdf8-dzzpr" podStartSLOduration=1.349569764 podStartE2EDuration="3.207935742s" totalImagesPullingTime="1.858365978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:58:08.206778156 +0000 UTC m=+111.347235223" watchObservedRunningTime="2026-07-06 23:58:08.207935742 +0000 UTC m=+111.348392799" Jul 06 23:58:15.908310 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:15.908274 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:15.908820 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:15.908341 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:15.912671 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:15.912650 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:16.207746 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:16.207724 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:58:16.260142 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:16.260115 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75c7867648-cxrs4"] Jul 06 23:58:17.212035 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:17.212003 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:58:41.280572 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.280528 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-75c7867648-cxrs4" podUID="d428aa6d-8373-4d21-b00e-96a38bef4716" containerName="console" containerID="cri-o://d13351da8d3723cdb287b2d2609d529cdd413549c6155a9bf261211da8a91e19" gracePeriod=15 Jul 06 23:58:41.559257 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.559183 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:58:41.675497 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.675464 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-console-config\") pod \"d428aa6d-8373-4d21-b00e-96a38bef4716\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " Jul 06 23:58:41.675497 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.675504 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-service-ca\") pod \"d428aa6d-8373-4d21-b00e-96a38bef4716\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " Jul 06 23:58:41.675751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.675572 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-oauth-serving-cert\") pod \"d428aa6d-8373-4d21-b00e-96a38bef4716\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " Jul 06 23:58:41.675751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.675596 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn648\" (UniqueName: \"kubernetes.io/projected/d428aa6d-8373-4d21-b00e-96a38bef4716-kube-api-access-kn648\") pod \"d428aa6d-8373-4d21-b00e-96a38bef4716\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " Jul 06 23:58:41.675751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.675627 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d428aa6d-8373-4d21-b00e-96a38bef4716-console-serving-cert\") pod \"d428aa6d-8373-4d21-b00e-96a38bef4716\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " Jul 06 23:58:41.675751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.675661 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d428aa6d-8373-4d21-b00e-96a38bef4716-console-oauth-config\") pod \"d428aa6d-8373-4d21-b00e-96a38bef4716\" (UID: \"d428aa6d-8373-4d21-b00e-96a38bef4716\") " Jul 06 23:58:41.675933 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.675903 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-console-config" (OuterVolumeSpecName: "console-config") pod "d428aa6d-8373-4d21-b00e-96a38bef4716" (UID: "d428aa6d-8373-4d21-b00e-96a38bef4716"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:41.675988 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.675950 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d428aa6d-8373-4d21-b00e-96a38bef4716" (UID: "d428aa6d-8373-4d21-b00e-96a38bef4716"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:41.676029 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.675986 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-service-ca" (OuterVolumeSpecName: "service-ca") pod "d428aa6d-8373-4d21-b00e-96a38bef4716" (UID: "d428aa6d-8373-4d21-b00e-96a38bef4716"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:41.677958 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.677922 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d428aa6d-8373-4d21-b00e-96a38bef4716-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d428aa6d-8373-4d21-b00e-96a38bef4716" (UID: "d428aa6d-8373-4d21-b00e-96a38bef4716"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:58:41.678070 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.677974 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d428aa6d-8373-4d21-b00e-96a38bef4716-kube-api-access-kn648" (OuterVolumeSpecName: "kube-api-access-kn648") pod "d428aa6d-8373-4d21-b00e-96a38bef4716" (UID: "d428aa6d-8373-4d21-b00e-96a38bef4716"). InnerVolumeSpecName "kube-api-access-kn648". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:58:41.678070 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.677973 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d428aa6d-8373-4d21-b00e-96a38bef4716-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d428aa6d-8373-4d21-b00e-96a38bef4716" (UID: "d428aa6d-8373-4d21-b00e-96a38bef4716"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:58:41.776723 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.776685 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-oauth-serving-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:58:41.776723 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.776716 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn648\" (UniqueName: \"kubernetes.io/projected/d428aa6d-8373-4d21-b00e-96a38bef4716-kube-api-access-kn648\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:58:41.776723 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.776726 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d428aa6d-8373-4d21-b00e-96a38bef4716-console-serving-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:58:41.776723 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.776736 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d428aa6d-8373-4d21-b00e-96a38bef4716-console-oauth-config\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:58:41.776995 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.776746 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-console-config\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:58:41.776995 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:41.776754 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d428aa6d-8373-4d21-b00e-96a38bef4716-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:58:42.297468 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:42.297413 2568 generic.go:356] "Generic (PLEG): container finished" podID="d428aa6d-8373-4d21-b00e-96a38bef4716" containerID="d13351da8d3723cdb287b2d2609d529cdd413549c6155a9bf261211da8a91e19" exitCode=0 Jul 06 23:58:42.297878 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:42.297483 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c7867648-cxrs4" Jul 06 23:58:42.297878 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:42.297519 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c7867648-cxrs4" event={"ID":"d428aa6d-8373-4d21-b00e-96a38bef4716","Type":"ContainerDied","Data":"d13351da8d3723cdb287b2d2609d529cdd413549c6155a9bf261211da8a91e19"} Jul 06 23:58:42.297878 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:42.297580 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c7867648-cxrs4" event={"ID":"d428aa6d-8373-4d21-b00e-96a38bef4716","Type":"ContainerDied","Data":"33bb0eb059d1851203d89a7564d2f8459759bbe94b46d9f82764cc414b664255"} Jul 06 23:58:42.297878 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:42.297603 2568 scope.go:117] "RemoveContainer" containerID="d13351da8d3723cdb287b2d2609d529cdd413549c6155a9bf261211da8a91e19" Jul 06 23:58:42.305767 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:42.305748 2568 scope.go:117] "RemoveContainer" containerID="d13351da8d3723cdb287b2d2609d529cdd413549c6155a9bf261211da8a91e19" Jul 06 23:58:42.306031 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:58:42.306011 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13351da8d3723cdb287b2d2609d529cdd413549c6155a9bf261211da8a91e19\": container with ID starting with d13351da8d3723cdb287b2d2609d529cdd413549c6155a9bf261211da8a91e19 not found: ID does not exist" containerID="d13351da8d3723cdb287b2d2609d529cdd413549c6155a9bf261211da8a91e19" Jul 06 23:58:42.306095 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:42.306039 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13351da8d3723cdb287b2d2609d529cdd413549c6155a9bf261211da8a91e19"} err="failed to get container status \"d13351da8d3723cdb287b2d2609d529cdd413549c6155a9bf261211da8a91e19\": rpc error: code = NotFound desc = could not find container \"d13351da8d3723cdb287b2d2609d529cdd413549c6155a9bf261211da8a91e19\": container with ID starting with d13351da8d3723cdb287b2d2609d529cdd413549c6155a9bf261211da8a91e19 not found: ID does not exist" Jul 06 23:58:42.321703 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:42.321649 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75c7867648-cxrs4"] Jul 06 23:58:42.327056 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:42.326992 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75c7867648-cxrs4"] Jul 06 23:58:43.484569 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:58:43.484536 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d428aa6d-8373-4d21-b00e-96a38bef4716" path="/var/lib/kubelet/pods/d428aa6d-8373-4d21-b00e-96a38bef4716/volumes" Jul 06 23:59:00.687804 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.687771 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54d54cffc5-wdhl4"] Jul 06 23:59:00.688245 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.688019 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d428aa6d-8373-4d21-b00e-96a38bef4716" containerName="console" Jul 06 23:59:00.688245 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.688031 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d428aa6d-8373-4d21-b00e-96a38bef4716" containerName="console" Jul 06 23:59:00.688245 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.688080 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d428aa6d-8373-4d21-b00e-96a38bef4716" containerName="console" Jul 06 23:59:00.690733 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.690708 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.709278 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.709254 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54d54cffc5-wdhl4"] Jul 06 23:59:00.806565 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.806537 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr424\" (UniqueName: \"kubernetes.io/projected/c5a75e2f-3431-4ba4-9e25-e560f629a67c-kube-api-access-zr424\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.806699 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.806575 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-trusted-ca-bundle\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.806699 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.806616 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-serving-cert\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.806699 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.806659 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-oauth-config\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.806699 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.806696 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-oauth-serving-cert\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.806881 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.806724 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-config\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.806881 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.806748 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-service-ca\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.907538 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.907511 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-config\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.907667 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.907549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-service-ca\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.907667 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.907607 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr424\" (UniqueName: \"kubernetes.io/projected/c5a75e2f-3431-4ba4-9e25-e560f629a67c-kube-api-access-zr424\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.907667 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.907639 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-trusted-ca-bundle\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.907827 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.907666 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-serving-cert\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.907827 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.907689 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-oauth-config\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.907827 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.907714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-oauth-serving-cert\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.908336 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.908309 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-config\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.908336 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.908328 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-service-ca\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.908497 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.908484 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-oauth-serving-cert\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.908543 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.908532 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-trusted-ca-bundle\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.910152 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.910127 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-serving-cert\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.910253 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.910227 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-oauth-config\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:00.915980 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:00.915956 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr424\" (UniqueName: \"kubernetes.io/projected/c5a75e2f-3431-4ba4-9e25-e560f629a67c-kube-api-access-zr424\") pod \"console-54d54cffc5-wdhl4\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:01.000081 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:01.000019 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:01.129923 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:01.129899 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54d54cffc5-wdhl4"] Jul 06 23:59:01.132166 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:59:01.132141 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a75e2f_3431_4ba4_9e25_e560f629a67c.slice/crio-17a81d1f54502a8339d4fb4389086acc6793ec07badb405eae103d3d6a218f08 WatchSource:0}: Error finding container 17a81d1f54502a8339d4fb4389086acc6793ec07badb405eae103d3d6a218f08: Status 404 returned error can't find the container with id 17a81d1f54502a8339d4fb4389086acc6793ec07badb405eae103d3d6a218f08 Jul 06 23:59:01.371228 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:01.371152 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d54cffc5-wdhl4" event={"ID":"c5a75e2f-3431-4ba4-9e25-e560f629a67c","Type":"ContainerStarted","Data":"56c8daf2c5ac1f5aee81b660a0908b1931f368ac491c5ce23ed69f28252c2390"} Jul 06 23:59:01.371228 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:01.371200 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d54cffc5-wdhl4" event={"ID":"c5a75e2f-3431-4ba4-9e25-e560f629a67c","Type":"ContainerStarted","Data":"17a81d1f54502a8339d4fb4389086acc6793ec07badb405eae103d3d6a218f08"} Jul 06 23:59:01.389714 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:01.389672 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54d54cffc5-wdhl4" podStartSLOduration=1.389658956 podStartE2EDuration="1.389658956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:59:01.389435572 +0000 UTC m=+164.529892641" watchObservedRunningTime="2026-07-06 23:59:01.389658956 +0000 UTC m=+164.530116012" Jul 06 23:59:11.000962 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:11.000884 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:11.000962 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:11.000934 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:11.005322 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:11.005302 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:11.408597 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:11.408518 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54d54cffc5-wdhl4" Jul 06 23:59:11.452110 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:11.452078 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c48c4fdf8-dzzpr"] Jul 06 23:59:36.473708 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.473642 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-c48c4fdf8-dzzpr" podUID="c38f5aa1-b6d5-4735-8cd0-d36d545185fd" containerName="console" containerID="cri-o://81c2fbf3a4f17311783d77da43eeec39edd8949d9d7a0e777de7185c6ecabe7b" gracePeriod=15 Jul 06 23:59:36.712196 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.712172 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:59:36.750753 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.750681 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-oauth-serving-cert\") pod \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " Jul 06 23:59:36.750753 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.750723 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-service-ca\") pod \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " Jul 06 23:59:36.750927 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.750762 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-serving-cert\") pod \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " Jul 06 23:59:36.750927 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.750787 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-config\") pod \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " Jul 06 23:59:36.750927 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.750826 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-oauth-config\") pod \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " Jul 06 23:59:36.750927 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.750853 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px97t\" (UniqueName: \"kubernetes.io/projected/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-kube-api-access-px97t\") pod \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " Jul 06 23:59:36.751112 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.750933 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-trusted-ca-bundle\") pod \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\" (UID: \"c38f5aa1-b6d5-4735-8cd0-d36d545185fd\") " Jul 06 23:59:36.751162 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.751102 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c38f5aa1-b6d5-4735-8cd0-d36d545185fd" (UID: "c38f5aa1-b6d5-4735-8cd0-d36d545185fd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:59:36.751162 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.751112 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-config" (OuterVolumeSpecName: "console-config") pod "c38f5aa1-b6d5-4735-8cd0-d36d545185fd" (UID: "c38f5aa1-b6d5-4735-8cd0-d36d545185fd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:59:36.751357 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.751331 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-service-ca" (OuterVolumeSpecName: "service-ca") pod "c38f5aa1-b6d5-4735-8cd0-d36d545185fd" (UID: "c38f5aa1-b6d5-4735-8cd0-d36d545185fd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:59:36.751478 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.751447 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c38f5aa1-b6d5-4735-8cd0-d36d545185fd" (UID: "c38f5aa1-b6d5-4735-8cd0-d36d545185fd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:59:36.752906 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.752882 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c38f5aa1-b6d5-4735-8cd0-d36d545185fd" (UID: "c38f5aa1-b6d5-4735-8cd0-d36d545185fd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:59:36.752988 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.752905 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c38f5aa1-b6d5-4735-8cd0-d36d545185fd" (UID: "c38f5aa1-b6d5-4735-8cd0-d36d545185fd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:59:36.752988 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.752965 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-kube-api-access-px97t" (OuterVolumeSpecName: "kube-api-access-px97t") pod "c38f5aa1-b6d5-4735-8cd0-d36d545185fd" (UID: "c38f5aa1-b6d5-4735-8cd0-d36d545185fd"). InnerVolumeSpecName "kube-api-access-px97t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:59:36.851712 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.851691 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-trusted-ca-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:59:36.851712 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.851711 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-oauth-serving-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:59:36.851867 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.851720 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:59:36.851867 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.851729 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-serving-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:59:36.851867 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.851738 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-config\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:59:36.851867 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.851747 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-console-oauth-config\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:59:36.851867 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:36.851755 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px97t\" (UniqueName: \"kubernetes.io/projected/c38f5aa1-b6d5-4735-8cd0-d36d545185fd-kube-api-access-px97t\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:59:37.499446 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:37.499401 2568 generic.go:356] "Generic (PLEG): container finished" podID="c38f5aa1-b6d5-4735-8cd0-d36d545185fd" containerID="81c2fbf3a4f17311783d77da43eeec39edd8949d9d7a0e777de7185c6ecabe7b" exitCode=0 Jul 06 23:59:37.499750 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:37.499448 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c48c4fdf8-dzzpr" event={"ID":"c38f5aa1-b6d5-4735-8cd0-d36d545185fd","Type":"ContainerDied","Data":"81c2fbf3a4f17311783d77da43eeec39edd8949d9d7a0e777de7185c6ecabe7b"} Jul 06 23:59:37.499750 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:37.499470 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c48c4fdf8-dzzpr" Jul 06 23:59:37.499750 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:37.499521 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c48c4fdf8-dzzpr" event={"ID":"c38f5aa1-b6d5-4735-8cd0-d36d545185fd","Type":"ContainerDied","Data":"6ad70a91a2062c6542628263a5e1a41d9c6ae111f075cd02d08717fba59cf974"} Jul 06 23:59:37.499750 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:37.499544 2568 scope.go:117] "RemoveContainer" containerID="81c2fbf3a4f17311783d77da43eeec39edd8949d9d7a0e777de7185c6ecabe7b" Jul 06 23:59:37.507864 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:37.507846 2568 scope.go:117] "RemoveContainer" containerID="81c2fbf3a4f17311783d77da43eeec39edd8949d9d7a0e777de7185c6ecabe7b" Jul 06 23:59:37.508112 ip-10-0-129-125 kubenswrapper[2568]: E0706 23:59:37.508096 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c2fbf3a4f17311783d77da43eeec39edd8949d9d7a0e777de7185c6ecabe7b\": container with ID starting with 81c2fbf3a4f17311783d77da43eeec39edd8949d9d7a0e777de7185c6ecabe7b not found: ID does not exist" containerID="81c2fbf3a4f17311783d77da43eeec39edd8949d9d7a0e777de7185c6ecabe7b" Jul 06 23:59:37.508176 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:37.508118 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c2fbf3a4f17311783d77da43eeec39edd8949d9d7a0e777de7185c6ecabe7b"} err="failed to get container status \"81c2fbf3a4f17311783d77da43eeec39edd8949d9d7a0e777de7185c6ecabe7b\": rpc error: code = NotFound desc = could not find container \"81c2fbf3a4f17311783d77da43eeec39edd8949d9d7a0e777de7185c6ecabe7b\": container with ID starting with 81c2fbf3a4f17311783d77da43eeec39edd8949d9d7a0e777de7185c6ecabe7b not found: ID does not exist" Jul 06 23:59:37.514780 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:37.514757 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c48c4fdf8-dzzpr"] Jul 06 23:59:37.518390 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:37.518367 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c48c4fdf8-dzzpr"] Jul 06 23:59:39.486080 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:39.486038 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c38f5aa1-b6d5-4735-8cd0-d36d545185fd" path="/var/lib/kubelet/pods/c38f5aa1-b6d5-4735-8cd0-d36d545185fd/volumes" Jul 06 23:59:42.810095 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.810064 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk"] Jul 06 23:59:42.810498 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.810306 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c38f5aa1-b6d5-4735-8cd0-d36d545185fd" containerName="console" Jul 06 23:59:42.810498 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.810316 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c38f5aa1-b6d5-4735-8cd0-d36d545185fd" containerName="console" Jul 06 23:59:42.810498 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.810362 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c38f5aa1-b6d5-4735-8cd0-d36d545185fd" containerName="console" Jul 06 23:59:42.812109 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.812092 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" Jul 06 23:59:42.814572 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.814552 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 23:59:42.814572 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.814552 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 23:59:42.815196 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.815183 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jkzb6\"" Jul 06 23:59:42.825721 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.825696 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk"] Jul 06 23:59:42.890579 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.890559 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd61976-4e2f-48ca-a6b4-01a37383810f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk\" (UID: \"7fd61976-4e2f-48ca-a6b4-01a37383810f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" Jul 06 23:59:42.890683 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.890597 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jl9\" (UniqueName: \"kubernetes.io/projected/7fd61976-4e2f-48ca-a6b4-01a37383810f-kube-api-access-q6jl9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk\" (UID: \"7fd61976-4e2f-48ca-a6b4-01a37383810f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" Jul 06 23:59:42.890683 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.890619 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd61976-4e2f-48ca-a6b4-01a37383810f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk\" (UID: \"7fd61976-4e2f-48ca-a6b4-01a37383810f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" Jul 06 23:59:42.991566 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.991546 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jl9\" (UniqueName: \"kubernetes.io/projected/7fd61976-4e2f-48ca-a6b4-01a37383810f-kube-api-access-q6jl9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk\" (UID: \"7fd61976-4e2f-48ca-a6b4-01a37383810f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" Jul 06 23:59:42.991651 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.991585 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd61976-4e2f-48ca-a6b4-01a37383810f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk\" (UID: \"7fd61976-4e2f-48ca-a6b4-01a37383810f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" Jul 06 23:59:42.991651 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.991627 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd61976-4e2f-48ca-a6b4-01a37383810f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk\" (UID: \"7fd61976-4e2f-48ca-a6b4-01a37383810f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" Jul 06 23:59:42.991997 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.991977 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd61976-4e2f-48ca-a6b4-01a37383810f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk\" (UID: \"7fd61976-4e2f-48ca-a6b4-01a37383810f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" Jul 06 23:59:42.992035 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:42.991993 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd61976-4e2f-48ca-a6b4-01a37383810f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk\" (UID: \"7fd61976-4e2f-48ca-a6b4-01a37383810f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" Jul 06 23:59:43.000254 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:43.000228 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jl9\" (UniqueName: \"kubernetes.io/projected/7fd61976-4e2f-48ca-a6b4-01a37383810f-kube-api-access-q6jl9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk\" (UID: \"7fd61976-4e2f-48ca-a6b4-01a37383810f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" Jul 06 23:59:43.120365 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:43.120311 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" Jul 06 23:59:43.248131 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:43.248099 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk"] Jul 06 23:59:43.251089 ip-10-0-129-125 kubenswrapper[2568]: W0706 23:59:43.251058 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd61976_4e2f_48ca_a6b4_01a37383810f.slice/crio-956c36243bf70d7b7905b1a8779395f8f77e48a8308c1ac10ac9c58bfc318970 WatchSource:0}: Error finding container 956c36243bf70d7b7905b1a8779395f8f77e48a8308c1ac10ac9c58bfc318970: Status 404 returned error can't find the container with id 956c36243bf70d7b7905b1a8779395f8f77e48a8308c1ac10ac9c58bfc318970 Jul 06 23:59:43.523442 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:43.523398 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" event={"ID":"7fd61976-4e2f-48ca-a6b4-01a37383810f","Type":"ContainerStarted","Data":"956c36243bf70d7b7905b1a8779395f8f77e48a8308c1ac10ac9c58bfc318970"} Jul 06 23:59:48.544791 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:48.544733 2568 generic.go:356] "Generic (PLEG): container finished" podID="7fd61976-4e2f-48ca-a6b4-01a37383810f" containerID="025a92ac03c1c1560012ddb554d57c4afa6b95abe5758c35d26bf825ce2cad95" exitCode=0 Jul 06 23:59:48.545122 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:48.544814 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" event={"ID":"7fd61976-4e2f-48ca-a6b4-01a37383810f","Type":"ContainerDied","Data":"025a92ac03c1c1560012ddb554d57c4afa6b95abe5758c35d26bf825ce2cad95"} Jul 06 23:59:50.553897 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:50.553867 2568 generic.go:356] "Generic (PLEG): container finished" podID="7fd61976-4e2f-48ca-a6b4-01a37383810f" containerID="087d050dcd5ff632a330c8d340034ce3de33c7ceb44d4941da27519190da6a90" exitCode=0 Jul 06 23:59:50.554166 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:50.553940 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" event={"ID":"7fd61976-4e2f-48ca-a6b4-01a37383810f","Type":"ContainerDied","Data":"087d050dcd5ff632a330c8d340034ce3de33c7ceb44d4941da27519190da6a90"} Jul 06 23:59:57.580880 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:57.580852 2568 generic.go:356] "Generic (PLEG): container finished" podID="7fd61976-4e2f-48ca-a6b4-01a37383810f" containerID="3ffe4216349a1c91099e56bda12c8444f486f1ccd263badd3d1b561c898b8322" exitCode=0 Jul 06 23:59:57.581235 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:57.580916 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" event={"ID":"7fd61976-4e2f-48ca-a6b4-01a37383810f","Type":"ContainerDied","Data":"3ffe4216349a1c91099e56bda12c8444f486f1ccd263badd3d1b561c898b8322"} Jul 06 23:59:58.709161 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:58.709139 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" Jul 06 23:59:58.803189 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:58.803164 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd61976-4e2f-48ca-a6b4-01a37383810f-util\") pod \"7fd61976-4e2f-48ca-a6b4-01a37383810f\" (UID: \"7fd61976-4e2f-48ca-a6b4-01a37383810f\") " Jul 06 23:59:58.803313 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:58.803220 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6jl9\" (UniqueName: \"kubernetes.io/projected/7fd61976-4e2f-48ca-a6b4-01a37383810f-kube-api-access-q6jl9\") pod \"7fd61976-4e2f-48ca-a6b4-01a37383810f\" (UID: \"7fd61976-4e2f-48ca-a6b4-01a37383810f\") " Jul 06 23:59:58.803313 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:58.803259 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd61976-4e2f-48ca-a6b4-01a37383810f-bundle\") pod \"7fd61976-4e2f-48ca-a6b4-01a37383810f\" (UID: \"7fd61976-4e2f-48ca-a6b4-01a37383810f\") " Jul 06 23:59:58.803977 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:58.803956 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd61976-4e2f-48ca-a6b4-01a37383810f-bundle" (OuterVolumeSpecName: "bundle") pod "7fd61976-4e2f-48ca-a6b4-01a37383810f" (UID: "7fd61976-4e2f-48ca-a6b4-01a37383810f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:59:58.805428 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:58.805398 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd61976-4e2f-48ca-a6b4-01a37383810f-kube-api-access-q6jl9" (OuterVolumeSpecName: "kube-api-access-q6jl9") pod "7fd61976-4e2f-48ca-a6b4-01a37383810f" (UID: "7fd61976-4e2f-48ca-a6b4-01a37383810f"). InnerVolumeSpecName "kube-api-access-q6jl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:59:58.807196 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:58.807175 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd61976-4e2f-48ca-a6b4-01a37383810f-util" (OuterVolumeSpecName: "util") pod "7fd61976-4e2f-48ca-a6b4-01a37383810f" (UID: "7fd61976-4e2f-48ca-a6b4-01a37383810f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:59:58.904304 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:58.904248 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6jl9\" (UniqueName: \"kubernetes.io/projected/7fd61976-4e2f-48ca-a6b4-01a37383810f-kube-api-access-q6jl9\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:59:58.904304 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:58.904270 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd61976-4e2f-48ca-a6b4-01a37383810f-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:59:58.904304 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:58.904283 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd61976-4e2f-48ca-a6b4-01a37383810f-util\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 06 23:59:59.588751 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:59.588719 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" event={"ID":"7fd61976-4e2f-48ca-a6b4-01a37383810f","Type":"ContainerDied","Data":"956c36243bf70d7b7905b1a8779395f8f77e48a8308c1ac10ac9c58bfc318970"} Jul 06 23:59:59.588859 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:59.588762 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754068xjk" Jul 06 23:59:59.588859 ip-10-0-129-125 kubenswrapper[2568]: I0706 23:59:59.588767 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="956c36243bf70d7b7905b1a8779395f8f77e48a8308c1ac10ac9c58bfc318970" Jul 07 00:00:00.172565 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.172538 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29723040-g55hm"] Jul 07 00:00:00.172905 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.172798 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd61976-4e2f-48ca-a6b4-01a37383810f" containerName="extract" Jul 07 00:00:00.172905 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.172808 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd61976-4e2f-48ca-a6b4-01a37383810f" containerName="extract" Jul 07 00:00:00.172905 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.172822 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd61976-4e2f-48ca-a6b4-01a37383810f" containerName="util" Jul 07 00:00:00.172905 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.172827 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd61976-4e2f-48ca-a6b4-01a37383810f" containerName="util" Jul 07 00:00:00.172905 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.172837 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd61976-4e2f-48ca-a6b4-01a37383810f" containerName="pull" Jul 07 00:00:00.172905 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.172842 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd61976-4e2f-48ca-a6b4-01a37383810f" containerName="pull" Jul 07 00:00:00.172905 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.172906 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd61976-4e2f-48ca-a6b4-01a37383810f" containerName="extract" Jul 07 00:00:00.175879 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.175858 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29723040-g55hm" Jul 07 00:00:00.183055 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.183035 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-sn2cq\"" Jul 07 00:00:00.183168 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.183101 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Jul 07 00:00:00.191522 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.191504 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29723040-g55hm"] Jul 07 00:00:00.211878 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.211857 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvt96\" (UniqueName: \"kubernetes.io/projected/27a6a626-55a6-4b1d-bca6-5377c676aae1-kube-api-access-tvt96\") pod \"image-pruner-29723040-g55hm\" (UID: \"27a6a626-55a6-4b1d-bca6-5377c676aae1\") " pod="openshift-image-registry/image-pruner-29723040-g55hm" Jul 07 00:00:00.211983 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.211892 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27a6a626-55a6-4b1d-bca6-5377c676aae1-serviceca\") pod \"image-pruner-29723040-g55hm\" (UID: \"27a6a626-55a6-4b1d-bca6-5377c676aae1\") " pod="openshift-image-registry/image-pruner-29723040-g55hm" Jul 07 00:00:00.312464 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.312445 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvt96\" (UniqueName: \"kubernetes.io/projected/27a6a626-55a6-4b1d-bca6-5377c676aae1-kube-api-access-tvt96\") pod \"image-pruner-29723040-g55hm\" (UID: \"27a6a626-55a6-4b1d-bca6-5377c676aae1\") " pod="openshift-image-registry/image-pruner-29723040-g55hm" Jul 07 00:00:00.312535 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.312485 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27a6a626-55a6-4b1d-bca6-5377c676aae1-serviceca\") pod \"image-pruner-29723040-g55hm\" (UID: \"27a6a626-55a6-4b1d-bca6-5377c676aae1\") " pod="openshift-image-registry/image-pruner-29723040-g55hm" Jul 07 00:00:00.313024 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.312996 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27a6a626-55a6-4b1d-bca6-5377c676aae1-serviceca\") pod \"image-pruner-29723040-g55hm\" (UID: \"27a6a626-55a6-4b1d-bca6-5377c676aae1\") " pod="openshift-image-registry/image-pruner-29723040-g55hm" Jul 07 00:00:00.321364 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.321341 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvt96\" (UniqueName: \"kubernetes.io/projected/27a6a626-55a6-4b1d-bca6-5377c676aae1-kube-api-access-tvt96\") pod \"image-pruner-29723040-g55hm\" (UID: \"27a6a626-55a6-4b1d-bca6-5377c676aae1\") " pod="openshift-image-registry/image-pruner-29723040-g55hm" Jul 07 00:00:00.510367 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.510347 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29723040-g55hm" Jul 07 00:00:00.638265 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:00.638239 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29723040-g55hm"] Jul 07 00:00:00.640963 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:00:00.640935 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a6a626_55a6_4b1d_bca6_5377c676aae1.slice/crio-56d8c516746b8f6aca9e7d1a6b913a46398ad481fb0545fa0587a23bf62cda11 WatchSource:0}: Error finding container 56d8c516746b8f6aca9e7d1a6b913a46398ad481fb0545fa0587a23bf62cda11: Status 404 returned error can't find the container with id 56d8c516746b8f6aca9e7d1a6b913a46398ad481fb0545fa0587a23bf62cda11 Jul 07 00:00:01.597700 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:01.597629 2568 generic.go:356] "Generic (PLEG): container finished" podID="27a6a626-55a6-4b1d-bca6-5377c676aae1" containerID="769dec56d2fe58f0f2418de72e32746c52d4daba30bfaa9c5ccc655cd821f6fd" exitCode=0 Jul 07 00:00:01.598011 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:01.597709 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29723040-g55hm" event={"ID":"27a6a626-55a6-4b1d-bca6-5377c676aae1","Type":"ContainerDied","Data":"769dec56d2fe58f0f2418de72e32746c52d4daba30bfaa9c5ccc655cd821f6fd"} Jul 07 00:00:01.598011 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:01.597751 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29723040-g55hm" event={"ID":"27a6a626-55a6-4b1d-bca6-5377c676aae1","Type":"ContainerStarted","Data":"56d8c516746b8f6aca9e7d1a6b913a46398ad481fb0545fa0587a23bf62cda11"} Jul 07 00:00:02.725771 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:02.725746 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29723040-g55hm" Jul 07 00:00:02.831239 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:02.831207 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27a6a626-55a6-4b1d-bca6-5377c676aae1-serviceca\") pod \"27a6a626-55a6-4b1d-bca6-5377c676aae1\" (UID: \"27a6a626-55a6-4b1d-bca6-5377c676aae1\") " Jul 07 00:00:02.831239 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:02.831244 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvt96\" (UniqueName: \"kubernetes.io/projected/27a6a626-55a6-4b1d-bca6-5377c676aae1-kube-api-access-tvt96\") pod \"27a6a626-55a6-4b1d-bca6-5377c676aae1\" (UID: \"27a6a626-55a6-4b1d-bca6-5377c676aae1\") " Jul 07 00:00:02.831538 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:02.831517 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a6a626-55a6-4b1d-bca6-5377c676aae1-serviceca" (OuterVolumeSpecName: "serviceca") pod "27a6a626-55a6-4b1d-bca6-5377c676aae1" (UID: "27a6a626-55a6-4b1d-bca6-5377c676aae1"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:00:02.833172 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:02.833149 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a6a626-55a6-4b1d-bca6-5377c676aae1-kube-api-access-tvt96" (OuterVolumeSpecName: "kube-api-access-tvt96") pod "27a6a626-55a6-4b1d-bca6-5377c676aae1" (UID: "27a6a626-55a6-4b1d-bca6-5377c676aae1"). InnerVolumeSpecName "kube-api-access-tvt96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:00:02.932641 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:02.932587 2568 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27a6a626-55a6-4b1d-bca6-5377c676aae1-serviceca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:00:02.932641 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:02.932615 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvt96\" (UniqueName: \"kubernetes.io/projected/27a6a626-55a6-4b1d-bca6-5377c676aae1-kube-api-access-tvt96\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:00:03.605569 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:03.605543 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29723040-g55hm" event={"ID":"27a6a626-55a6-4b1d-bca6-5377c676aae1","Type":"ContainerDied","Data":"56d8c516746b8f6aca9e7d1a6b913a46398ad481fb0545fa0587a23bf62cda11"} Jul 07 00:00:03.605686 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:03.605588 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56d8c516746b8f6aca9e7d1a6b913a46398ad481fb0545fa0587a23bf62cda11" Jul 07 00:00:03.605686 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:03.605555 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29723040-g55hm" Jul 07 00:00:06.035977 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.035945 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2nbz7"] Jul 07 00:00:06.036355 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.036202 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a6a626-55a6-4b1d-bca6-5377c676aae1" containerName="image-pruner" Jul 07 00:00:06.036355 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.036213 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a6a626-55a6-4b1d-bca6-5377c676aae1" containerName="image-pruner" Jul 07 00:00:06.036355 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.036260 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a6a626-55a6-4b1d-bca6-5377c676aae1" containerName="image-pruner" Jul 07 00:00:06.039037 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.039022 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2nbz7" Jul 07 00:00:06.041573 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.041548 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-f92b8\"" Jul 07 00:00:06.041717 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.041590 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 07 00:00:06.041717 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.041604 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 07 00:00:06.051358 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.051332 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2nbz7"] Jul 07 00:00:06.151391 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.151364 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhl67\" (UniqueName: \"kubernetes.io/projected/31ebd3ab-d123-4ea7-8561-36724442c2f2-kube-api-access-zhl67\") pod \"cert-manager-operator-controller-manager-55c444bf86-2nbz7\" (UID: \"31ebd3ab-d123-4ea7-8561-36724442c2f2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2nbz7" Jul 07 00:00:06.151521 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.151434 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31ebd3ab-d123-4ea7-8561-36724442c2f2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2nbz7\" (UID: \"31ebd3ab-d123-4ea7-8561-36724442c2f2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2nbz7" Jul 07 00:00:06.252312 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.252289 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31ebd3ab-d123-4ea7-8561-36724442c2f2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2nbz7\" (UID: \"31ebd3ab-d123-4ea7-8561-36724442c2f2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2nbz7" Jul 07 00:00:06.252415 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.252332 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhl67\" (UniqueName: \"kubernetes.io/projected/31ebd3ab-d123-4ea7-8561-36724442c2f2-kube-api-access-zhl67\") pod \"cert-manager-operator-controller-manager-55c444bf86-2nbz7\" (UID: \"31ebd3ab-d123-4ea7-8561-36724442c2f2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2nbz7" Jul 07 00:00:06.252706 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.252690 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31ebd3ab-d123-4ea7-8561-36724442c2f2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2nbz7\" (UID: \"31ebd3ab-d123-4ea7-8561-36724442c2f2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2nbz7" Jul 07 00:00:06.261133 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.261110 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhl67\" (UniqueName: \"kubernetes.io/projected/31ebd3ab-d123-4ea7-8561-36724442c2f2-kube-api-access-zhl67\") pod \"cert-manager-operator-controller-manager-55c444bf86-2nbz7\" (UID: \"31ebd3ab-d123-4ea7-8561-36724442c2f2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2nbz7" Jul 07 00:00:06.347552 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.347501 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2nbz7" Jul 07 00:00:06.494060 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.494039 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2nbz7"] Jul 07 00:00:06.496921 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:00:06.496896 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ebd3ab_d123_4ea7_8561_36724442c2f2.slice/crio-8363835302c9fb41475c1802484276b8bcd2413d57e117b0c0a54138812979b8 WatchSource:0}: Error finding container 8363835302c9fb41475c1802484276b8bcd2413d57e117b0c0a54138812979b8: Status 404 returned error can't find the container with id 8363835302c9fb41475c1802484276b8bcd2413d57e117b0c0a54138812979b8 Jul 07 00:00:06.619641 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:06.619588 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2nbz7" event={"ID":"31ebd3ab-d123-4ea7-8561-36724442c2f2","Type":"ContainerStarted","Data":"8363835302c9fb41475c1802484276b8bcd2413d57e117b0c0a54138812979b8"} Jul 07 00:00:18.664959 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:18.664921 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2nbz7" event={"ID":"31ebd3ab-d123-4ea7-8561-36724442c2f2","Type":"ContainerStarted","Data":"822aa6183ac12943cd7b66cb80615ab7bffc1274979b2b0328fc571f94a6efbd"} Jul 07 00:00:18.693566 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:18.693513 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2nbz7" podStartSLOduration=1.342125581 podStartE2EDuration="12.693494151s" totalImagesPullingTime="11.35136857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:00:18.691166743 +0000 UTC m=+241.831623797" watchObservedRunningTime="2026-07-07 00:00:18.693494151 +0000 UTC m=+241.833951207" Jul 07 00:00:19.470700 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.470669 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr"] Jul 07 00:00:19.474059 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.474042 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" Jul 07 00:00:19.476585 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.476560 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 00:00:19.476696 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.476597 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 00:00:19.477467 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.477454 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jkzb6\"" Jul 07 00:00:19.488321 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.488298 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr"] Jul 07 00:00:19.541836 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.541816 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f01a21c5-e9af-48f0-841c-6f62f615e51d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr\" (UID: \"f01a21c5-e9af-48f0-841c-6f62f615e51d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" Jul 07 00:00:19.541937 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.541846 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01a21c5-e9af-48f0-841c-6f62f615e51d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr\" (UID: \"f01a21c5-e9af-48f0-841c-6f62f615e51d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" Jul 07 00:00:19.541937 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.541893 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhqjm\" (UniqueName: \"kubernetes.io/projected/f01a21c5-e9af-48f0-841c-6f62f615e51d-kube-api-access-vhqjm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr\" (UID: \"f01a21c5-e9af-48f0-841c-6f62f615e51d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" Jul 07 00:00:19.642754 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.642732 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhqjm\" (UniqueName: \"kubernetes.io/projected/f01a21c5-e9af-48f0-841c-6f62f615e51d-kube-api-access-vhqjm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr\" (UID: \"f01a21c5-e9af-48f0-841c-6f62f615e51d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" Jul 07 00:00:19.642873 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.642780 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f01a21c5-e9af-48f0-841c-6f62f615e51d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr\" (UID: \"f01a21c5-e9af-48f0-841c-6f62f615e51d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" Jul 07 00:00:19.642873 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.642812 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01a21c5-e9af-48f0-841c-6f62f615e51d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr\" (UID: \"f01a21c5-e9af-48f0-841c-6f62f615e51d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" Jul 07 00:00:19.643083 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.643065 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f01a21c5-e9af-48f0-841c-6f62f615e51d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr\" (UID: \"f01a21c5-e9af-48f0-841c-6f62f615e51d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" Jul 07 00:00:19.643150 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.643130 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01a21c5-e9af-48f0-841c-6f62f615e51d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr\" (UID: \"f01a21c5-e9af-48f0-841c-6f62f615e51d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" Jul 07 00:00:19.651567 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.651546 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhqjm\" (UniqueName: \"kubernetes.io/projected/f01a21c5-e9af-48f0-841c-6f62f615e51d-kube-api-access-vhqjm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr\" (UID: \"f01a21c5-e9af-48f0-841c-6f62f615e51d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" Jul 07 00:00:19.783490 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.783444 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" Jul 07 00:00:19.911505 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:19.911469 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr"] Jul 07 00:00:19.913435 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:00:19.913388 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01a21c5_e9af_48f0_841c_6f62f615e51d.slice/crio-cb2a7748cbbd53eff3ec3b54e8d76e2f5c55eafb71eaffb6991dd6400515e1ff WatchSource:0}: Error finding container cb2a7748cbbd53eff3ec3b54e8d76e2f5c55eafb71eaffb6991dd6400515e1ff: Status 404 returned error can't find the container with id cb2a7748cbbd53eff3ec3b54e8d76e2f5c55eafb71eaffb6991dd6400515e1ff Jul 07 00:00:20.673251 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:20.673219 2568 generic.go:356] "Generic (PLEG): container finished" podID="f01a21c5-e9af-48f0-841c-6f62f615e51d" containerID="689c80bbfb728accae9bd454b7cf8da2b6f7639bb99e2d773613217a157a1bf2" exitCode=0 Jul 07 00:00:20.673377 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:20.673271 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" event={"ID":"f01a21c5-e9af-48f0-841c-6f62f615e51d","Type":"ContainerDied","Data":"689c80bbfb728accae9bd454b7cf8da2b6f7639bb99e2d773613217a157a1bf2"} Jul 07 00:00:20.673377 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:20.673320 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" event={"ID":"f01a21c5-e9af-48f0-841c-6f62f615e51d","Type":"ContainerStarted","Data":"cb2a7748cbbd53eff3ec3b54e8d76e2f5c55eafb71eaffb6991dd6400515e1ff"} Jul 07 00:00:29.295843 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:29.295809 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k85s9"] Jul 07 00:00:29.338860 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:29.338835 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k85s9"] Jul 07 00:00:29.338997 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:29.338915 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-k85s9" Jul 07 00:00:29.342131 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:29.342108 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 00:00:29.343045 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:29.343020 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 00:00:29.343045 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:29.343034 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hgd99\"" Jul 07 00:00:29.404278 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:29.404250 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxh5v\" (UniqueName: \"kubernetes.io/projected/8f604d99-34ef-4a6e-92ac-714d9043bc3e-kube-api-access-fxh5v\") pod \"cert-manager-55b4d954d5-k85s9\" (UID: \"8f604d99-34ef-4a6e-92ac-714d9043bc3e\") " pod="cert-manager/cert-manager-55b4d954d5-k85s9" Jul 07 00:00:29.404367 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:29.404290 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f604d99-34ef-4a6e-92ac-714d9043bc3e-bound-sa-token\") pod \"cert-manager-55b4d954d5-k85s9\" (UID: \"8f604d99-34ef-4a6e-92ac-714d9043bc3e\") " pod="cert-manager/cert-manager-55b4d954d5-k85s9" Jul 07 00:00:29.504810 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:29.504785 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxh5v\" (UniqueName: \"kubernetes.io/projected/8f604d99-34ef-4a6e-92ac-714d9043bc3e-kube-api-access-fxh5v\") pod \"cert-manager-55b4d954d5-k85s9\" (UID: \"8f604d99-34ef-4a6e-92ac-714d9043bc3e\") " pod="cert-manager/cert-manager-55b4d954d5-k85s9" Jul 07 00:00:29.504896 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:29.504821 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f604d99-34ef-4a6e-92ac-714d9043bc3e-bound-sa-token\") pod \"cert-manager-55b4d954d5-k85s9\" (UID: \"8f604d99-34ef-4a6e-92ac-714d9043bc3e\") " pod="cert-manager/cert-manager-55b4d954d5-k85s9" Jul 07 00:00:29.513207 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:29.513187 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxh5v\" (UniqueName: \"kubernetes.io/projected/8f604d99-34ef-4a6e-92ac-714d9043bc3e-kube-api-access-fxh5v\") pod \"cert-manager-55b4d954d5-k85s9\" (UID: \"8f604d99-34ef-4a6e-92ac-714d9043bc3e\") " pod="cert-manager/cert-manager-55b4d954d5-k85s9" Jul 07 00:00:29.513294 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:29.513225 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f604d99-34ef-4a6e-92ac-714d9043bc3e-bound-sa-token\") pod \"cert-manager-55b4d954d5-k85s9\" (UID: \"8f604d99-34ef-4a6e-92ac-714d9043bc3e\") " pod="cert-manager/cert-manager-55b4d954d5-k85s9" Jul 07 00:00:29.648251 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:29.648202 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-k85s9" Jul 07 00:00:29.793953 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:29.793924 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k85s9"] Jul 07 00:00:29.796967 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:00:29.796931 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f604d99_34ef_4a6e_92ac_714d9043bc3e.slice/crio-72ffd3524937a3a8cd748fbbb5e2efe8811dd1e49a00f0ea6e8c321860f4e9d6 WatchSource:0}: Error finding container 72ffd3524937a3a8cd748fbbb5e2efe8811dd1e49a00f0ea6e8c321860f4e9d6: Status 404 returned error can't find the container with id 72ffd3524937a3a8cd748fbbb5e2efe8811dd1e49a00f0ea6e8c321860f4e9d6 Jul 07 00:00:30.711743 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:30.711711 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-k85s9" event={"ID":"8f604d99-34ef-4a6e-92ac-714d9043bc3e","Type":"ContainerStarted","Data":"72ffd3524937a3a8cd748fbbb5e2efe8811dd1e49a00f0ea6e8c321860f4e9d6"} Jul 07 00:00:33.726289 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:33.726254 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-k85s9" event={"ID":"8f604d99-34ef-4a6e-92ac-714d9043bc3e","Type":"ContainerStarted","Data":"b8ffbc195b0bda1be0701472516d7afc279d7f24b19b62e8eee766c588ff135e"} Jul 07 00:00:33.741027 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:33.740985 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-k85s9" podStartSLOduration=1.48436774 podStartE2EDuration="4.740971734s" totalImagesPullingTime="3.256603994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:00:33.740383673 +0000 UTC m=+256.880840776" watchObservedRunningTime="2026-07-07 00:00:33.740971734 +0000 UTC m=+256.881428790" Jul 07 00:00:49.786012 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:49.785973 2568 generic.go:356] "Generic (PLEG): container finished" podID="f01a21c5-e9af-48f0-841c-6f62f615e51d" containerID="4fb84c6de8da080876dace1224560035d7d00aa9a9cf569d7b7d47e05e315f5e" exitCode=0 Jul 07 00:00:49.786414 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:49.786060 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" event={"ID":"f01a21c5-e9af-48f0-841c-6f62f615e51d","Type":"ContainerDied","Data":"4fb84c6de8da080876dace1224560035d7d00aa9a9cf569d7b7d47e05e315f5e"} Jul 07 00:00:50.792513 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:50.792478 2568 generic.go:356] "Generic (PLEG): container finished" podID="f01a21c5-e9af-48f0-841c-6f62f615e51d" containerID="f94ae3e91f0b7785982a6a4c30eae89fc97220eaddf7f474dd6dd7c7053ba3c4" exitCode=0 Jul 07 00:00:50.792880 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:50.792527 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" event={"ID":"f01a21c5-e9af-48f0-841c-6f62f615e51d","Type":"ContainerDied","Data":"f94ae3e91f0b7785982a6a4c30eae89fc97220eaddf7f474dd6dd7c7053ba3c4"} Jul 07 00:00:51.921894 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:51.921867 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" Jul 07 00:00:52.055263 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:52.055190 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhqjm\" (UniqueName: \"kubernetes.io/projected/f01a21c5-e9af-48f0-841c-6f62f615e51d-kube-api-access-vhqjm\") pod \"f01a21c5-e9af-48f0-841c-6f62f615e51d\" (UID: \"f01a21c5-e9af-48f0-841c-6f62f615e51d\") " Jul 07 00:00:52.055263 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:52.055225 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f01a21c5-e9af-48f0-841c-6f62f615e51d-util\") pod \"f01a21c5-e9af-48f0-841c-6f62f615e51d\" (UID: \"f01a21c5-e9af-48f0-841c-6f62f615e51d\") " Jul 07 00:00:52.055263 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:52.055252 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01a21c5-e9af-48f0-841c-6f62f615e51d-bundle\") pod \"f01a21c5-e9af-48f0-841c-6f62f615e51d\" (UID: \"f01a21c5-e9af-48f0-841c-6f62f615e51d\") " Jul 07 00:00:52.055722 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:52.055661 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01a21c5-e9af-48f0-841c-6f62f615e51d-bundle" (OuterVolumeSpecName: "bundle") pod "f01a21c5-e9af-48f0-841c-6f62f615e51d" (UID: "f01a21c5-e9af-48f0-841c-6f62f615e51d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:00:52.057285 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:52.057264 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01a21c5-e9af-48f0-841c-6f62f615e51d-kube-api-access-vhqjm" (OuterVolumeSpecName: "kube-api-access-vhqjm") pod "f01a21c5-e9af-48f0-841c-6f62f615e51d" (UID: "f01a21c5-e9af-48f0-841c-6f62f615e51d"). InnerVolumeSpecName "kube-api-access-vhqjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:00:52.059981 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:52.059962 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01a21c5-e9af-48f0-841c-6f62f615e51d-util" (OuterVolumeSpecName: "util") pod "f01a21c5-e9af-48f0-841c-6f62f615e51d" (UID: "f01a21c5-e9af-48f0-841c-6f62f615e51d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:00:52.156630 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:52.156605 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhqjm\" (UniqueName: \"kubernetes.io/projected/f01a21c5-e9af-48f0-841c-6f62f615e51d-kube-api-access-vhqjm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:00:52.156630 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:52.156624 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f01a21c5-e9af-48f0-841c-6f62f615e51d-util\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:00:52.156630 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:52.156633 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01a21c5-e9af-48f0-841c-6f62f615e51d-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:00:52.801696 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:52.801660 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" event={"ID":"f01a21c5-e9af-48f0-841c-6f62f615e51d","Type":"ContainerDied","Data":"cb2a7748cbbd53eff3ec3b54e8d76e2f5c55eafb71eaffb6991dd6400515e1ff"} Jul 07 00:00:52.801876 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:52.801706 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb2a7748cbbd53eff3ec3b54e8d76e2f5c55eafb71eaffb6991dd6400515e1ff" Jul 07 00:00:52.801876 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:52.801707 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f24fdr" Jul 07 00:00:56.077089 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.077057 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pd4f"] Jul 07 00:00:56.077548 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.077333 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f01a21c5-e9af-48f0-841c-6f62f615e51d" containerName="extract" Jul 07 00:00:56.077548 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.077343 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f01a21c5-e9af-48f0-841c-6f62f615e51d" containerName="extract" Jul 07 00:00:56.077548 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.077357 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f01a21c5-e9af-48f0-841c-6f62f615e51d" containerName="util" Jul 07 00:00:56.077548 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.077362 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f01a21c5-e9af-48f0-841c-6f62f615e51d" containerName="util" Jul 07 00:00:56.077548 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.077368 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f01a21c5-e9af-48f0-841c-6f62f615e51d" containerName="pull" Jul 07 00:00:56.077548 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.077375 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f01a21c5-e9af-48f0-841c-6f62f615e51d" containerName="pull" Jul 07 00:00:56.077548 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.077433 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f01a21c5-e9af-48f0-841c-6f62f615e51d" containerName="extract" Jul 07 00:00:56.080258 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.080240 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pd4f" Jul 07 00:00:56.082651 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.082626 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-d4shl\"" Jul 07 00:00:56.083434 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.083399 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 00:00:56.083527 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.083455 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 00:00:56.088312 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.088289 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pd4f"] Jul 07 00:00:56.185031 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.185008 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgw9q\" (UniqueName: \"kubernetes.io/projected/6a94c09d-7726-40ba-8c4f-af646ba02179-kube-api-access-zgw9q\") pod \"openshift-lws-operator-fd8ccff4c-6pd4f\" (UID: \"6a94c09d-7726-40ba-8c4f-af646ba02179\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pd4f" Jul 07 00:00:56.185127 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.185044 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a94c09d-7726-40ba-8c4f-af646ba02179-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6pd4f\" (UID: \"6a94c09d-7726-40ba-8c4f-af646ba02179\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pd4f" Jul 07 00:00:56.285459 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.285414 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a94c09d-7726-40ba-8c4f-af646ba02179-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6pd4f\" (UID: \"6a94c09d-7726-40ba-8c4f-af646ba02179\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pd4f" Jul 07 00:00:56.285549 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.285524 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgw9q\" (UniqueName: \"kubernetes.io/projected/6a94c09d-7726-40ba-8c4f-af646ba02179-kube-api-access-zgw9q\") pod \"openshift-lws-operator-fd8ccff4c-6pd4f\" (UID: \"6a94c09d-7726-40ba-8c4f-af646ba02179\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pd4f" Jul 07 00:00:56.285753 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.285734 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a94c09d-7726-40ba-8c4f-af646ba02179-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6pd4f\" (UID: \"6a94c09d-7726-40ba-8c4f-af646ba02179\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pd4f" Jul 07 00:00:56.294236 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.294217 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgw9q\" (UniqueName: \"kubernetes.io/projected/6a94c09d-7726-40ba-8c4f-af646ba02179-kube-api-access-zgw9q\") pod \"openshift-lws-operator-fd8ccff4c-6pd4f\" (UID: \"6a94c09d-7726-40ba-8c4f-af646ba02179\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pd4f" Jul 07 00:00:56.389054 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.389002 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pd4f" Jul 07 00:00:56.515959 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.515929 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pd4f"] Jul 07 00:00:56.519158 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:00:56.519131 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a94c09d_7726_40ba_8c4f_af646ba02179.slice/crio-31b26c846eb88a70c40227360eca6c6dd9d99e35013e8926c0a73f5c01874922 WatchSource:0}: Error finding container 31b26c846eb88a70c40227360eca6c6dd9d99e35013e8926c0a73f5c01874922: Status 404 returned error can't find the container with id 31b26c846eb88a70c40227360eca6c6dd9d99e35013e8926c0a73f5c01874922 Jul 07 00:00:56.819511 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:00:56.819480 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pd4f" event={"ID":"6a94c09d-7726-40ba-8c4f-af646ba02179","Type":"ContainerStarted","Data":"31b26c846eb88a70c40227360eca6c6dd9d99e35013e8926c0a73f5c01874922"} Jul 07 00:01:07.863648 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:07.863606 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pd4f" event={"ID":"6a94c09d-7726-40ba-8c4f-af646ba02179","Type":"ContainerStarted","Data":"513d6ab9a769f35ee5e4035824762bf17a131beca93823ff7a8f8f35de46d994"} Jul 07 00:01:07.881703 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:07.881653 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pd4f" podStartSLOduration=1.509496168 podStartE2EDuration="11.88163961s" totalImagesPullingTime="10.372143442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:01:07.879976709 +0000 UTC m=+291.020433765" watchObservedRunningTime="2026-07-07 00:01:07.88163961 +0000 UTC m=+291.022096666" Jul 07 00:01:09.527074 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.527034 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6"] Jul 07 00:01:09.529272 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.529255 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" Jul 07 00:01:09.531636 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.531614 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 00:01:09.531781 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.531614 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 00:01:09.531781 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.531659 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jkzb6\"" Jul 07 00:01:09.539647 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.539621 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6"] Jul 07 00:01:09.693738 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.693698 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6\" (UID: \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" Jul 07 00:01:09.693925 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.693752 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6\" (UID: \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" Jul 07 00:01:09.693925 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.693817 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n54b\" (UniqueName: \"kubernetes.io/projected/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-kube-api-access-9n54b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6\" (UID: \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" Jul 07 00:01:09.795250 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.795141 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6\" (UID: \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" Jul 07 00:01:09.795250 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.795209 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n54b\" (UniqueName: \"kubernetes.io/projected/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-kube-api-access-9n54b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6\" (UID: \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" Jul 07 00:01:09.795547 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.795277 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6\" (UID: \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" Jul 07 00:01:09.795610 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.795576 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6\" (UID: \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" Jul 07 00:01:09.795688 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.795666 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6\" (UID: \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" Jul 07 00:01:09.803835 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.803816 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n54b\" (UniqueName: \"kubernetes.io/projected/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-kube-api-access-9n54b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6\" (UID: \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" Jul 07 00:01:09.838717 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.838688 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" Jul 07 00:01:09.967345 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:09.967311 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6"] Jul 07 00:01:09.970761 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:01:09.970731 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0bc362_90b6_4ef4_af7a_0b264682a8eb.slice/crio-696a4c014ece0aa29428e1c3f1f8b34808cbf222a4b036f947111d2b2db6549b WatchSource:0}: Error finding container 696a4c014ece0aa29428e1c3f1f8b34808cbf222a4b036f947111d2b2db6549b: Status 404 returned error can't find the container with id 696a4c014ece0aa29428e1c3f1f8b34808cbf222a4b036f947111d2b2db6549b Jul 07 00:01:10.878351 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:10.878319 2568 generic.go:356] "Generic (PLEG): container finished" podID="7a0bc362-90b6-4ef4-af7a-0b264682a8eb" containerID="1d043a030fa65b12725c63d89143521f74c4d772fb55b08355074c0dcab5ca11" exitCode=0 Jul 07 00:01:10.878728 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:10.878357 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" event={"ID":"7a0bc362-90b6-4ef4-af7a-0b264682a8eb","Type":"ContainerDied","Data":"1d043a030fa65b12725c63d89143521f74c4d772fb55b08355074c0dcab5ca11"} Jul 07 00:01:10.878728 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:10.878392 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" event={"ID":"7a0bc362-90b6-4ef4-af7a-0b264682a8eb","Type":"ContainerStarted","Data":"696a4c014ece0aa29428e1c3f1f8b34808cbf222a4b036f947111d2b2db6549b"} Jul 07 00:01:11.883837 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:11.883804 2568 generic.go:356] "Generic (PLEG): container finished" podID="7a0bc362-90b6-4ef4-af7a-0b264682a8eb" containerID="5d4ad20b7df9156ebe26a3b25896f796bf46eec9c66ad3f7073aecdadbf4dc97" exitCode=0 Jul 07 00:01:11.884257 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:11.883875 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" event={"ID":"7a0bc362-90b6-4ef4-af7a-0b264682a8eb","Type":"ContainerDied","Data":"5d4ad20b7df9156ebe26a3b25896f796bf46eec9c66ad3f7073aecdadbf4dc97"} Jul 07 00:01:12.890701 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:12.890660 2568 generic.go:356] "Generic (PLEG): container finished" podID="7a0bc362-90b6-4ef4-af7a-0b264682a8eb" containerID="16035cb1e37dc830f35fd5228b9c843e9786ec69c2104ac87241ab0075fb548b" exitCode=0 Jul 07 00:01:12.890701 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:12.890692 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" event={"ID":"7a0bc362-90b6-4ef4-af7a-0b264682a8eb","Type":"ContainerDied","Data":"16035cb1e37dc830f35fd5228b9c843e9786ec69c2104ac87241ab0075fb548b"} Jul 07 00:01:14.019945 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.019917 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" Jul 07 00:01:14.129160 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.129127 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-bundle\") pod \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\" (UID: \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\") " Jul 07 00:01:14.129347 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.129221 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n54b\" (UniqueName: \"kubernetes.io/projected/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-kube-api-access-9n54b\") pod \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\" (UID: \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\") " Jul 07 00:01:14.129347 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.129261 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-util\") pod \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\" (UID: \"7a0bc362-90b6-4ef4-af7a-0b264682a8eb\") " Jul 07 00:01:14.129938 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.129908 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-bundle" (OuterVolumeSpecName: "bundle") pod "7a0bc362-90b6-4ef4-af7a-0b264682a8eb" (UID: "7a0bc362-90b6-4ef4-af7a-0b264682a8eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:01:14.131262 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.131236 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-kube-api-access-9n54b" (OuterVolumeSpecName: "kube-api-access-9n54b") pod "7a0bc362-90b6-4ef4-af7a-0b264682a8eb" (UID: "7a0bc362-90b6-4ef4-af7a-0b264682a8eb"). InnerVolumeSpecName "kube-api-access-9n54b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:01:14.134813 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.134777 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-util" (OuterVolumeSpecName: "util") pod "7a0bc362-90b6-4ef4-af7a-0b264682a8eb" (UID: "7a0bc362-90b6-4ef4-af7a-0b264682a8eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:01:14.230749 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.230715 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n54b\" (UniqueName: \"kubernetes.io/projected/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-kube-api-access-9n54b\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:01:14.230749 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.230746 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-util\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:01:14.230749 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.230755 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a0bc362-90b6-4ef4-af7a-0b264682a8eb-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:01:14.902502 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.902465 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" event={"ID":"7a0bc362-90b6-4ef4-af7a-0b264682a8eb","Type":"ContainerDied","Data":"696a4c014ece0aa29428e1c3f1f8b34808cbf222a4b036f947111d2b2db6549b"} Jul 07 00:01:14.902698 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.902522 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="696a4c014ece0aa29428e1c3f1f8b34808cbf222a4b036f947111d2b2db6549b" Jul 07 00:01:14.902698 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.902482 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8kp6" Jul 07 00:01:14.992396 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.992364 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4"] Jul 07 00:01:14.992662 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.992650 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a0bc362-90b6-4ef4-af7a-0b264682a8eb" containerName="extract" Jul 07 00:01:14.992704 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.992664 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a0bc362-90b6-4ef4-af7a-0b264682a8eb" containerName="extract" Jul 07 00:01:14.992704 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.992681 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a0bc362-90b6-4ef4-af7a-0b264682a8eb" containerName="util" Jul 07 00:01:14.992704 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.992688 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a0bc362-90b6-4ef4-af7a-0b264682a8eb" containerName="util" Jul 07 00:01:14.992704 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.992694 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a0bc362-90b6-4ef4-af7a-0b264682a8eb" containerName="pull" Jul 07 00:01:14.992704 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.992701 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a0bc362-90b6-4ef4-af7a-0b264682a8eb" containerName="pull" Jul 07 00:01:14.992845 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.992750 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a0bc362-90b6-4ef4-af7a-0b264682a8eb" containerName="extract" Jul 07 00:01:14.994492 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.994477 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:14.996769 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.996751 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 00:01:14.996932 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.996915 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 00:01:14.996986 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.996940 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4s9zb\"" Jul 07 00:01:14.997534 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:14.997519 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 00:01:15.004182 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.004159 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4"] Jul 07 00:01:15.139179 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.139143 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a2aeb0b-b009-46bf-bf6b-4013269603b6-cert\") pod \"lws-controller-manager-5c45554c7d-zplz4\" (UID: \"9a2aeb0b-b009-46bf-bf6b-4013269603b6\") " pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:15.139587 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.139205 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p94dq\" (UniqueName: \"kubernetes.io/projected/9a2aeb0b-b009-46bf-bf6b-4013269603b6-kube-api-access-p94dq\") pod \"lws-controller-manager-5c45554c7d-zplz4\" (UID: \"9a2aeb0b-b009-46bf-bf6b-4013269603b6\") " pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:15.139587 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.139266 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a2aeb0b-b009-46bf-bf6b-4013269603b6-metrics-cert\") pod \"lws-controller-manager-5c45554c7d-zplz4\" (UID: \"9a2aeb0b-b009-46bf-bf6b-4013269603b6\") " pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:15.139587 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.139288 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9a2aeb0b-b009-46bf-bf6b-4013269603b6-manager-config\") pod \"lws-controller-manager-5c45554c7d-zplz4\" (UID: \"9a2aeb0b-b009-46bf-bf6b-4013269603b6\") " pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:15.239932 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.239899 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p94dq\" (UniqueName: \"kubernetes.io/projected/9a2aeb0b-b009-46bf-bf6b-4013269603b6-kube-api-access-p94dq\") pod \"lws-controller-manager-5c45554c7d-zplz4\" (UID: \"9a2aeb0b-b009-46bf-bf6b-4013269603b6\") " pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:15.240103 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.239953 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a2aeb0b-b009-46bf-bf6b-4013269603b6-metrics-cert\") pod \"lws-controller-manager-5c45554c7d-zplz4\" (UID: \"9a2aeb0b-b009-46bf-bf6b-4013269603b6\") " pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:15.240103 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.239979 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9a2aeb0b-b009-46bf-bf6b-4013269603b6-manager-config\") pod \"lws-controller-manager-5c45554c7d-zplz4\" (UID: \"9a2aeb0b-b009-46bf-bf6b-4013269603b6\") " pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:15.240103 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.240004 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a2aeb0b-b009-46bf-bf6b-4013269603b6-cert\") pod \"lws-controller-manager-5c45554c7d-zplz4\" (UID: \"9a2aeb0b-b009-46bf-bf6b-4013269603b6\") " pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:15.240782 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.240758 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9a2aeb0b-b009-46bf-bf6b-4013269603b6-manager-config\") pod \"lws-controller-manager-5c45554c7d-zplz4\" (UID: \"9a2aeb0b-b009-46bf-bf6b-4013269603b6\") " pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:15.242510 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.242484 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a2aeb0b-b009-46bf-bf6b-4013269603b6-metrics-cert\") pod \"lws-controller-manager-5c45554c7d-zplz4\" (UID: \"9a2aeb0b-b009-46bf-bf6b-4013269603b6\") " pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:15.242587 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.242541 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a2aeb0b-b009-46bf-bf6b-4013269603b6-cert\") pod \"lws-controller-manager-5c45554c7d-zplz4\" (UID: \"9a2aeb0b-b009-46bf-bf6b-4013269603b6\") " pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:15.248242 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.248222 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p94dq\" (UniqueName: \"kubernetes.io/projected/9a2aeb0b-b009-46bf-bf6b-4013269603b6-kube-api-access-p94dq\") pod \"lws-controller-manager-5c45554c7d-zplz4\" (UID: \"9a2aeb0b-b009-46bf-bf6b-4013269603b6\") " pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:15.304523 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.304485 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:15.428168 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.428114 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4"] Jul 07 00:01:15.430300 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:01:15.430278 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2aeb0b_b009_46bf_bf6b_4013269603b6.slice/crio-df744b08399b4cb37ae7ac3361621f45aeae1a5bd640f92465b8b730838b2305 WatchSource:0}: Error finding container df744b08399b4cb37ae7ac3361621f45aeae1a5bd640f92465b8b730838b2305: Status 404 returned error can't find the container with id df744b08399b4cb37ae7ac3361621f45aeae1a5bd640f92465b8b730838b2305 Jul 07 00:01:15.908049 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:15.908009 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" event={"ID":"9a2aeb0b-b009-46bf-bf6b-4013269603b6","Type":"ContainerStarted","Data":"df744b08399b4cb37ae7ac3361621f45aeae1a5bd640f92465b8b730838b2305"} Jul 07 00:01:16.913703 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:16.913603 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" event={"ID":"9a2aeb0b-b009-46bf-bf6b-4013269603b6","Type":"ContainerStarted","Data":"a1c4a0c8cb5b4e4552fd0cfbf4af672797cf4f11c65e1430fb3f7c12bfef8041"} Jul 07 00:01:16.913703 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:16.913675 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:16.947511 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:16.947450 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" podStartSLOduration=1.721671922 podStartE2EDuration="2.947432486s" totalImagesPullingTime="1.225760564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:01:16.946559436 +0000 UTC m=+300.087016493" watchObservedRunningTime="2026-07-07 00:01:16.947432486 +0000 UTC m=+300.087889540" Jul 07 00:01:17.351720 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:17.351694 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 00:01:26.783436 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:26.783391 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm"] Jul 07 00:01:26.789727 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:26.789711 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:26.793774 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:26.793751 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 00:01:26.793970 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:26.793955 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 00:01:26.794253 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:26.794236 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 00:01:26.794318 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:26.794266 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jl66k\"" Jul 07 00:01:26.794376 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:26.794340 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 00:01:26.805730 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:26.805710 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm"] Jul 07 00:01:26.926233 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:26.926205 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7091db00-88eb-40b5-91a1-6ca17af6a084-manifests\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:26.926349 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:26.926242 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7091db00-88eb-40b5-91a1-6ca17af6a084-tmp\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:26.926349 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:26.926269 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7091db00-88eb-40b5-91a1-6ca17af6a084-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:26.926349 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:26.926313 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7091db00-88eb-40b5-91a1-6ca17af6a084-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:26.926472 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:26.926357 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k2bp\" (UniqueName: \"kubernetes.io/projected/7091db00-88eb-40b5-91a1-6ca17af6a084-kube-api-access-7k2bp\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:27.027554 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.027527 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7091db00-88eb-40b5-91a1-6ca17af6a084-tmp\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:27.027671 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.027562 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7091db00-88eb-40b5-91a1-6ca17af6a084-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:27.027671 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.027594 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7091db00-88eb-40b5-91a1-6ca17af6a084-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:27.027671 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.027649 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k2bp\" (UniqueName: \"kubernetes.io/projected/7091db00-88eb-40b5-91a1-6ca17af6a084-kube-api-access-7k2bp\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:27.027799 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.027694 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7091db00-88eb-40b5-91a1-6ca17af6a084-manifests\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:27.028044 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.028015 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7091db00-88eb-40b5-91a1-6ca17af6a084-manifests\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:27.029827 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.029806 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7091db00-88eb-40b5-91a1-6ca17af6a084-tmp\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:27.030196 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.030175 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7091db00-88eb-40b5-91a1-6ca17af6a084-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:27.030288 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.030268 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7091db00-88eb-40b5-91a1-6ca17af6a084-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:27.037468 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.037397 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k2bp\" (UniqueName: \"kubernetes.io/projected/7091db00-88eb-40b5-91a1-6ca17af6a084-kube-api-access-7k2bp\") pod \"opendatahub-operator-controller-manager-fcdfc77c6-wqnxm\" (UID: \"7091db00-88eb-40b5-91a1-6ca17af6a084\") " pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:27.100741 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.100721 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:27.234623 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.234594 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm"] Jul 07 00:01:27.240097 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:01:27.240075 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7091db00_88eb_40b5_91a1_6ca17af6a084.slice/crio-3d57f715a006f11d15893a047b8a0d72188a7ca8604f6524488413bae1096519 WatchSource:0}: Error finding container 3d57f715a006f11d15893a047b8a0d72188a7ca8604f6524488413bae1096519: Status 404 returned error can't find the container with id 3d57f715a006f11d15893a047b8a0d72188a7ca8604f6524488413bae1096519 Jul 07 00:01:27.242486 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.242466 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 00:01:27.920137 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.920110 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c45554c7d-zplz4" Jul 07 00:01:27.962478 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:27.962442 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" event={"ID":"7091db00-88eb-40b5-91a1-6ca17af6a084","Type":"ContainerStarted","Data":"3d57f715a006f11d15893a047b8a0d72188a7ca8604f6524488413bae1096519"} Jul 07 00:01:30.978878 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:30.978847 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" event={"ID":"7091db00-88eb-40b5-91a1-6ca17af6a084","Type":"ContainerStarted","Data":"76c6863683c1864e28ae16a30235f92b6e5ae8f02a6e1b81bac668dcb403972e"} Jul 07 00:01:31.986861 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:31.986827 2568 generic.go:356] "Generic (PLEG): container finished" podID="7091db00-88eb-40b5-91a1-6ca17af6a084" containerID="76c6863683c1864e28ae16a30235f92b6e5ae8f02a6e1b81bac668dcb403972e" exitCode=0 Jul 07 00:01:31.987245 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:31.986911 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" event={"ID":"7091db00-88eb-40b5-91a1-6ca17af6a084","Type":"ContainerDied","Data":"76c6863683c1864e28ae16a30235f92b6e5ae8f02a6e1b81bac668dcb403972e"} Jul 07 00:01:33.998018 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:33.997976 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" event={"ID":"7091db00-88eb-40b5-91a1-6ca17af6a084","Type":"ContainerStarted","Data":"808690f42f4fd0e29ef48ee050332288937c881ba88bebe7b2be9807b6b679a9"} Jul 07 00:01:33.998409 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:33.998102 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:34.036170 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:34.036119 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" podStartSLOduration=2.3789992890000002 podStartE2EDuration="8.036102804s" totalImagesPullingTime="4.585952893s" totalInitContainerRuntime="1.071150622s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:01:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:01:34.035866739 +0000 UTC m=+317.176323812" watchObservedRunningTime="2026-07-07 00:01:34.036102804 +0000 UTC m=+317.176559857" Jul 07 00:01:45.004776 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:45.004746 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-fcdfc77c6-wqnxm" Jul 07 00:01:55.093868 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.093840 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf"] Jul 07 00:01:55.099608 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.099590 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.103083 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.103056 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 00:01:55.103209 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.103145 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 00:01:55.103445 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.103406 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 00:01:55.103693 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.103671 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zqbz7\"" Jul 07 00:01:55.122904 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.122883 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf"] Jul 07 00:01:55.135392 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.135369 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.135516 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.135406 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-istio-data\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.135516 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.135445 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.135615 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.135512 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f07e276-b83e-4532-95d1-ce7221280785-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.135615 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.135551 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfv8d\" (UniqueName: \"kubernetes.io/projected/5f07e276-b83e-4532-95d1-ce7221280785-kube-api-access-dfv8d\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.135701 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.135631 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.135701 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.135660 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f07e276-b83e-4532-95d1-ce7221280785-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.135701 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.135683 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.135816 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.135724 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f07e276-b83e-4532-95d1-ce7221280785-istio-token\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.236036 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.236012 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.236134 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.236049 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f07e276-b83e-4532-95d1-ce7221280785-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.236134 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.236071 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfv8d\" (UniqueName: \"kubernetes.io/projected/5f07e276-b83e-4532-95d1-ce7221280785-kube-api-access-dfv8d\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.236134 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.236110 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.236301 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.236280 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f07e276-b83e-4532-95d1-ce7221280785-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.236368 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.236331 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.236446 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.236407 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f07e276-b83e-4532-95d1-ce7221280785-istio-token\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.236507 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.236479 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.236565 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.236537 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-istio-data\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.236692 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.236668 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.236880 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.236861 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-istio-data\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.236946 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.236927 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.237054 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.237036 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.237091 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.237075 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f07e276-b83e-4532-95d1-ce7221280785-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.238357 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.238330 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.238588 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.238572 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f07e276-b83e-4532-95d1-ce7221280785-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.252260 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.252239 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfv8d\" (UniqueName: \"kubernetes.io/projected/5f07e276-b83e-4532-95d1-ce7221280785-kube-api-access-dfv8d\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.257568 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.257537 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f07e276-b83e-4532-95d1-ce7221280785-istio-token\") pod \"data-science-gateway-data-science-gateway-class-548557588-n7pjf\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.308730 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.308708 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p"] Jul 07 00:01:55.313980 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.313966 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.316599 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.316577 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 00:01:55.316702 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.316579 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 00:01:55.316702 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.316581 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 00:01:55.317948 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.317932 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-78jhc\"" Jul 07 00:01:55.327682 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.327662 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p"] Jul 07 00:01:55.337681 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.337662 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c75d750-3b58-42ff-9951-d40d64c8c75f-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.337779 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.337694 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c75d750-3b58-42ff-9951-d40d64c8c75f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.337779 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.337720 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c75d750-3b58-42ff-9951-d40d64c8c75f-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.337899 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.337801 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c75d750-3b58-42ff-9951-d40d64c8c75f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.337899 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.337848 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c75d750-3b58-42ff-9951-d40d64c8c75f-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.337985 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.337895 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txtj5\" (UniqueName: \"kubernetes.io/projected/9c75d750-3b58-42ff-9951-d40d64c8c75f-kube-api-access-txtj5\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.337985 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.337930 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c75d750-3b58-42ff-9951-d40d64c8c75f-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.409804 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.409755 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:01:55.439090 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.439065 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c75d750-3b58-42ff-9951-d40d64c8c75f-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.439183 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.439110 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txtj5\" (UniqueName: \"kubernetes.io/projected/9c75d750-3b58-42ff-9951-d40d64c8c75f-kube-api-access-txtj5\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.439183 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.439138 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c75d750-3b58-42ff-9951-d40d64c8c75f-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.439183 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.439166 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c75d750-3b58-42ff-9951-d40d64c8c75f-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.439317 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.439207 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c75d750-3b58-42ff-9951-d40d64c8c75f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.439317 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.439243 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c75d750-3b58-42ff-9951-d40d64c8c75f-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.439541 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.439465 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c75d750-3b58-42ff-9951-d40d64c8c75f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.440122 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.440085 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c75d750-3b58-42ff-9951-d40d64c8c75f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.441228 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.441207 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c75d750-3b58-42ff-9951-d40d64c8c75f-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.441919 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.441901 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c75d750-3b58-42ff-9951-d40d64c8c75f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.441995 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.441957 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c75d750-3b58-42ff-9951-d40d64c8c75f-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.442081 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.442059 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c75d750-3b58-42ff-9951-d40d64c8c75f-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.447159 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.447136 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txtj5\" (UniqueName: \"kubernetes.io/projected/9c75d750-3b58-42ff-9951-d40d64c8c75f-kube-api-access-txtj5\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.447229 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.447197 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c75d750-3b58-42ff-9951-d40d64c8c75f-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9br2p\" (UID: \"9c75d750-3b58-42ff-9951-d40d64c8c75f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.554573 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.554541 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf"] Jul 07 00:01:55.556243 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:01:55.556221 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f07e276_b83e_4532_95d1_ce7221280785.slice/crio-d2bb37ec399b7c3d331376026e3419ca00a5421cb20c4a5f911e2c49cc0c0773 WatchSource:0}: Error finding container d2bb37ec399b7c3d331376026e3419ca00a5421cb20c4a5f911e2c49cc0c0773: Status 404 returned error can't find the container with id d2bb37ec399b7c3d331376026e3419ca00a5421cb20c4a5f911e2c49cc0c0773 Jul 07 00:01:55.623630 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.623605 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:01:55.771260 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:55.771217 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p"] Jul 07 00:01:55.773246 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:01:55.773223 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c75d750_3b58_42ff_9951_d40d64c8c75f.slice/crio-84f60c47ad53c6ef4d565fb8b94475be5c242b1123224804ce0c8052b532bb43 WatchSource:0}: Error finding container 84f60c47ad53c6ef4d565fb8b94475be5c242b1123224804ce0c8052b532bb43: Status 404 returned error can't find the container with id 84f60c47ad53c6ef4d565fb8b94475be5c242b1123224804ce0c8052b532bb43 Jul 07 00:01:56.095699 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:56.095663 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" event={"ID":"9c75d750-3b58-42ff-9951-d40d64c8c75f","Type":"ContainerStarted","Data":"84f60c47ad53c6ef4d565fb8b94475be5c242b1123224804ce0c8052b532bb43"} Jul 07 00:01:56.096708 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:56.096684 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" event={"ID":"5f07e276-b83e-4532-95d1-ce7221280785","Type":"ContainerStarted","Data":"d2bb37ec399b7c3d331376026e3419ca00a5421cb20c4a5f911e2c49cc0c0773"} Jul 07 00:01:59.284718 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:59.284677 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:01:59.285093 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:59.284755 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:01:59.285093 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:59.284783 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:01:59.286743 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:59.286721 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:01:59.286813 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:01:59.286798 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:02:00.120673 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:00.120618 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" event={"ID":"9c75d750-3b58-42ff-9951-d40d64c8c75f","Type":"ContainerStarted","Data":"14bde83767931c5b8adeae1b960b127a233eced238311e2c050fa527fc1c71db"} Jul 07 00:02:00.120937 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:00.120916 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:02:00.122540 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:00.122504 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" event={"ID":"5f07e276-b83e-4532-95d1-ce7221280785","Type":"ContainerStarted","Data":"60eca9dccaccf897ea0bd1ce8544f6e02bdf5ff91e56a6879033bd0e183e0daf"} Jul 07 00:02:00.122754 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:00.122721 2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-9br2p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 00:02:00.122861 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:00.122831 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" podUID="9c75d750-3b58-42ff-9951-d40d64c8c75f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 00:02:00.149672 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:00.149627 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" podStartSLOduration=1.638247214 podStartE2EDuration="5.149613802s" totalImagesPullingTime="3.511366588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:02:00.145884987 +0000 UTC m=+343.286342044" watchObservedRunningTime="2026-07-07 00:02:00.149613802 +0000 UTC m=+343.290070858" Jul 07 00:02:00.178355 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:00.178313 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" podStartSLOduration=1.4521480740000001 podStartE2EDuration="5.178300442s" totalImagesPullingTime="3.726152368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:02:00.175949744 +0000 UTC m=+343.316406800" watchObservedRunningTime="2026-07-07 00:02:00.178300442 +0000 UTC m=+343.318757499" Jul 07 00:02:00.410819 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:00.410752 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:02:00.411798 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:00.411778 2568 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-548557588-n7pjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 07 00:02:00.411852 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:00.411824 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" podUID="5f07e276-b83e-4532-95d1-ce7221280785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 07 00:02:01.127912 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:01.127883 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9br2p" Jul 07 00:02:01.411164 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:01.411091 2568 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-548557588-n7pjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 07 00:02:01.411164 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:01.411143 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" podUID="5f07e276-b83e-4532-95d1-ce7221280785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 07 00:02:02.411207 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:02.411171 2568 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-548557588-n7pjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 07 00:02:02.411608 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:02.411224 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" podUID="5f07e276-b83e-4532-95d1-ce7221280785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 07 00:02:02.684722 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:02.684641 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf"] Jul 07 00:02:03.136574 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:03.136510 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" podUID="5f07e276-b83e-4532-95d1-ce7221280785" containerName="istio-proxy" containerID="cri-o://60eca9dccaccf897ea0bd1ce8544f6e02bdf5ff91e56a6879033bd0e183e0daf" gracePeriod=30 Jul 07 00:02:08.370594 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.370570 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:02:08.448646 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.448621 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f07e276-b83e-4532-95d1-ce7221280785-istio-podinfo\") pod \"5f07e276-b83e-4532-95d1-ce7221280785\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " Jul 07 00:02:08.448764 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.448661 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-workload-certs\") pod \"5f07e276-b83e-4532-95d1-ce7221280785\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " Jul 07 00:02:08.448764 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.448690 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-istio-data\") pod \"5f07e276-b83e-4532-95d1-ce7221280785\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " Jul 07 00:02:08.448764 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.448719 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfv8d\" (UniqueName: \"kubernetes.io/projected/5f07e276-b83e-4532-95d1-ce7221280785-kube-api-access-dfv8d\") pod \"5f07e276-b83e-4532-95d1-ce7221280785\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " Jul 07 00:02:08.448764 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.448755 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-credential-socket\") pod \"5f07e276-b83e-4532-95d1-ce7221280785\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " Jul 07 00:02:08.448961 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.448777 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f07e276-b83e-4532-95d1-ce7221280785-istiod-ca-cert\") pod \"5f07e276-b83e-4532-95d1-ce7221280785\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " Jul 07 00:02:08.448961 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.448811 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f07e276-b83e-4532-95d1-ce7221280785-istio-token\") pod \"5f07e276-b83e-4532-95d1-ce7221280785\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " Jul 07 00:02:08.448961 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.448849 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-istio-envoy\") pod \"5f07e276-b83e-4532-95d1-ce7221280785\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " Jul 07 00:02:08.448961 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.448874 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-workload-socket\") pod \"5f07e276-b83e-4532-95d1-ce7221280785\" (UID: \"5f07e276-b83e-4532-95d1-ce7221280785\") " Jul 07 00:02:08.449148 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.449032 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f07e276-b83e-4532-95d1-ce7221280785" (UID: "5f07e276-b83e-4532-95d1-ce7221280785"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:08.449148 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.449039 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f07e276-b83e-4532-95d1-ce7221280785" (UID: "5f07e276-b83e-4532-95d1-ce7221280785"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:08.449148 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.449085 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f07e276-b83e-4532-95d1-ce7221280785" (UID: "5f07e276-b83e-4532-95d1-ce7221280785"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:08.449148 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.449119 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-workload-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:02:08.449148 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.449137 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-istio-data\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:02:08.449352 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.449251 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f07e276-b83e-4532-95d1-ce7221280785" (UID: "5f07e276-b83e-4532-95d1-ce7221280785"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:08.449485 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.449464 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f07e276-b83e-4532-95d1-ce7221280785-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f07e276-b83e-4532-95d1-ce7221280785" (UID: "5f07e276-b83e-4532-95d1-ce7221280785"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:02:08.451071 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.451044 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f07e276-b83e-4532-95d1-ce7221280785-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f07e276-b83e-4532-95d1-ce7221280785" (UID: "5f07e276-b83e-4532-95d1-ce7221280785"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:02:08.451071 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.451056 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f07e276-b83e-4532-95d1-ce7221280785" (UID: "5f07e276-b83e-4532-95d1-ce7221280785"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:08.451198 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.451181 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f07e276-b83e-4532-95d1-ce7221280785-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f07e276-b83e-4532-95d1-ce7221280785" (UID: "5f07e276-b83e-4532-95d1-ce7221280785"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:02:08.451198 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.451188 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f07e276-b83e-4532-95d1-ce7221280785-kube-api-access-dfv8d" (OuterVolumeSpecName: "kube-api-access-dfv8d") pod "5f07e276-b83e-4532-95d1-ce7221280785" (UID: "5f07e276-b83e-4532-95d1-ce7221280785"). InnerVolumeSpecName "kube-api-access-dfv8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:02:08.549838 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.549808 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f07e276-b83e-4532-95d1-ce7221280785-istio-token\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:02:08.549838 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.549834 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-istio-envoy\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:02:08.549992 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.549847 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-workload-socket\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:02:08.549992 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.549859 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f07e276-b83e-4532-95d1-ce7221280785-istio-podinfo\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:02:08.549992 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.549871 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfv8d\" (UniqueName: \"kubernetes.io/projected/5f07e276-b83e-4532-95d1-ce7221280785-kube-api-access-dfv8d\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:02:08.549992 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.549883 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07e276-b83e-4532-95d1-ce7221280785-credential-socket\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:02:08.549992 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:08.549894 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f07e276-b83e-4532-95d1-ce7221280785-istiod-ca-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:02:09.163700 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:09.163629 2568 generic.go:356] "Generic (PLEG): container finished" podID="5f07e276-b83e-4532-95d1-ce7221280785" containerID="60eca9dccaccf897ea0bd1ce8544f6e02bdf5ff91e56a6879033bd0e183e0daf" exitCode=0 Jul 07 00:02:09.163700 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:09.163668 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" event={"ID":"5f07e276-b83e-4532-95d1-ce7221280785","Type":"ContainerDied","Data":"60eca9dccaccf897ea0bd1ce8544f6e02bdf5ff91e56a6879033bd0e183e0daf"} Jul 07 00:02:09.163700 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:09.163674 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" Jul 07 00:02:09.163930 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:09.163714 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf" event={"ID":"5f07e276-b83e-4532-95d1-ce7221280785","Type":"ContainerDied","Data":"d2bb37ec399b7c3d331376026e3419ca00a5421cb20c4a5f911e2c49cc0c0773"} Jul 07 00:02:09.163930 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:09.163741 2568 scope.go:117] "RemoveContainer" containerID="60eca9dccaccf897ea0bd1ce8544f6e02bdf5ff91e56a6879033bd0e183e0daf" Jul 07 00:02:09.172375 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:09.172354 2568 scope.go:117] "RemoveContainer" containerID="60eca9dccaccf897ea0bd1ce8544f6e02bdf5ff91e56a6879033bd0e183e0daf" Jul 07 00:02:09.172658 ip-10-0-129-125 kubenswrapper[2568]: E0707 00:02:09.172636 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60eca9dccaccf897ea0bd1ce8544f6e02bdf5ff91e56a6879033bd0e183e0daf\": container with ID starting with 60eca9dccaccf897ea0bd1ce8544f6e02bdf5ff91e56a6879033bd0e183e0daf not found: ID does not exist" containerID="60eca9dccaccf897ea0bd1ce8544f6e02bdf5ff91e56a6879033bd0e183e0daf" Jul 07 00:02:09.172719 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:09.172671 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60eca9dccaccf897ea0bd1ce8544f6e02bdf5ff91e56a6879033bd0e183e0daf"} err="failed to get container status \"60eca9dccaccf897ea0bd1ce8544f6e02bdf5ff91e56a6879033bd0e183e0daf\": rpc error: code = NotFound desc = could not find container \"60eca9dccaccf897ea0bd1ce8544f6e02bdf5ff91e56a6879033bd0e183e0daf\": container with ID starting with 60eca9dccaccf897ea0bd1ce8544f6e02bdf5ff91e56a6879033bd0e183e0daf not found: ID does not exist" Jul 07 00:02:09.184906 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:09.184882 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf"] Jul 07 00:02:09.188523 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:09.188503 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-n7pjf"] Jul 07 00:02:09.485676 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:02:09.485640 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f07e276-b83e-4532-95d1-ce7221280785" path="/var/lib/kubelet/pods/5f07e276-b83e-4532-95d1-ce7221280785/volumes" Jul 07 00:03:03.951299 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:03.951268 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl"] Jul 07 00:03:03.951796 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:03.951585 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f07e276-b83e-4532-95d1-ce7221280785" containerName="istio-proxy" Jul 07 00:03:03.951796 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:03.951597 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f07e276-b83e-4532-95d1-ce7221280785" containerName="istio-proxy" Jul 07 00:03:03.951796 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:03.951665 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f07e276-b83e-4532-95d1-ce7221280785" containerName="istio-proxy" Jul 07 00:03:03.953923 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:03.953907 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" Jul 07 00:03:03.957038 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:03.957010 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4nstc\"" Jul 07 00:03:03.957160 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:03.957053 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 00:03:03.957585 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:03.957554 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 00:03:03.964819 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:03.964801 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl"] Jul 07 00:03:03.973141 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:03.973120 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9872f2e0-1bdc-41d4-bd52-250eb055a764-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl\" (UID: \"9872f2e0-1bdc-41d4-bd52-250eb055a764\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" Jul 07 00:03:03.973237 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:03.973174 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9872f2e0-1bdc-41d4-bd52-250eb055a764-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl\" (UID: \"9872f2e0-1bdc-41d4-bd52-250eb055a764\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" Jul 07 00:03:03.973293 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:03.973267 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqk8h\" (UniqueName: \"kubernetes.io/projected/9872f2e0-1bdc-41d4-bd52-250eb055a764-kube-api-access-zqk8h\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl\" (UID: \"9872f2e0-1bdc-41d4-bd52-250eb055a764\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" Jul 07 00:03:04.074312 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.074280 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9872f2e0-1bdc-41d4-bd52-250eb055a764-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl\" (UID: \"9872f2e0-1bdc-41d4-bd52-250eb055a764\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" Jul 07 00:03:04.074521 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.074325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqk8h\" (UniqueName: \"kubernetes.io/projected/9872f2e0-1bdc-41d4-bd52-250eb055a764-kube-api-access-zqk8h\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl\" (UID: \"9872f2e0-1bdc-41d4-bd52-250eb055a764\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" Jul 07 00:03:04.074521 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.074367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9872f2e0-1bdc-41d4-bd52-250eb055a764-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl\" (UID: \"9872f2e0-1bdc-41d4-bd52-250eb055a764\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" Jul 07 00:03:04.074691 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.074669 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9872f2e0-1bdc-41d4-bd52-250eb055a764-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl\" (UID: \"9872f2e0-1bdc-41d4-bd52-250eb055a764\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" Jul 07 00:03:04.074752 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.074700 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9872f2e0-1bdc-41d4-bd52-250eb055a764-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl\" (UID: \"9872f2e0-1bdc-41d4-bd52-250eb055a764\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" Jul 07 00:03:04.083239 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.083217 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqk8h\" (UniqueName: \"kubernetes.io/projected/9872f2e0-1bdc-41d4-bd52-250eb055a764-kube-api-access-zqk8h\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl\" (UID: \"9872f2e0-1bdc-41d4-bd52-250eb055a764\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" Jul 07 00:03:04.182777 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.182747 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl"] Jul 07 00:03:04.185140 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.185126 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" Jul 07 00:03:04.246533 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.246456 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl"] Jul 07 00:03:04.262655 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.262623 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" Jul 07 00:03:04.276102 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.276075 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1a71ff1-2865-4800-8b89-8c62a56ee213-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl\" (UID: \"c1a71ff1-2865-4800-8b89-8c62a56ee213\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" Jul 07 00:03:04.276241 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.276111 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1a71ff1-2865-4800-8b89-8c62a56ee213-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl\" (UID: \"c1a71ff1-2865-4800-8b89-8c62a56ee213\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" Jul 07 00:03:04.276241 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.276209 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmxgv\" (UniqueName: \"kubernetes.io/projected/c1a71ff1-2865-4800-8b89-8c62a56ee213-kube-api-access-wmxgv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl\" (UID: \"c1a71ff1-2865-4800-8b89-8c62a56ee213\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" Jul 07 00:03:04.377078 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.377048 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmxgv\" (UniqueName: \"kubernetes.io/projected/c1a71ff1-2865-4800-8b89-8c62a56ee213-kube-api-access-wmxgv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl\" (UID: \"c1a71ff1-2865-4800-8b89-8c62a56ee213\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" Jul 07 00:03:04.377237 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.377099 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1a71ff1-2865-4800-8b89-8c62a56ee213-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl\" (UID: \"c1a71ff1-2865-4800-8b89-8c62a56ee213\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" Jul 07 00:03:04.377237 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.377124 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1a71ff1-2865-4800-8b89-8c62a56ee213-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl\" (UID: \"c1a71ff1-2865-4800-8b89-8c62a56ee213\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" Jul 07 00:03:04.377627 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.377605 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1a71ff1-2865-4800-8b89-8c62a56ee213-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl\" (UID: \"c1a71ff1-2865-4800-8b89-8c62a56ee213\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" Jul 07 00:03:04.377703 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.377616 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1a71ff1-2865-4800-8b89-8c62a56ee213-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl\" (UID: \"c1a71ff1-2865-4800-8b89-8c62a56ee213\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" Jul 07 00:03:04.392054 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.392029 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmxgv\" (UniqueName: \"kubernetes.io/projected/c1a71ff1-2865-4800-8b89-8c62a56ee213-kube-api-access-wmxgv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl\" (UID: \"c1a71ff1-2865-4800-8b89-8c62a56ee213\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" Jul 07 00:03:04.419128 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:03:04.419104 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9872f2e0_1bdc_41d4_bd52_250eb055a764.slice/crio-2c02cd59fc42a02a1ac09a1cd9fc709118cedcdf007199f1664b865715fee3af WatchSource:0}: Error finding container 2c02cd59fc42a02a1ac09a1cd9fc709118cedcdf007199f1664b865715fee3af: Status 404 returned error can't find the container with id 2c02cd59fc42a02a1ac09a1cd9fc709118cedcdf007199f1664b865715fee3af Jul 07 00:03:04.424014 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.423989 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl"] Jul 07 00:03:04.482464 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.482437 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5"] Jul 07 00:03:04.484625 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.484609 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" Jul 07 00:03:04.494007 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.493986 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" Jul 07 00:03:04.504593 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.504571 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5"] Jul 07 00:03:04.578942 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.578914 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93feb81c-29f0-48d8-8001-f00293c545dc-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5\" (UID: \"93feb81c-29f0-48d8-8001-f00293c545dc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" Jul 07 00:03:04.579063 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.579026 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93feb81c-29f0-48d8-8001-f00293c545dc-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5\" (UID: \"93feb81c-29f0-48d8-8001-f00293c545dc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" Jul 07 00:03:04.579129 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.579071 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l99hd\" (UniqueName: \"kubernetes.io/projected/93feb81c-29f0-48d8-8001-f00293c545dc-kube-api-access-l99hd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5\" (UID: \"93feb81c-29f0-48d8-8001-f00293c545dc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" Jul 07 00:03:04.622927 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.622903 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl"] Jul 07 00:03:04.624380 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:03:04.624356 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a71ff1_2865_4800_8b89_8c62a56ee213.slice/crio-0564e0d30507010d77c27b370595114e0d9f86e0f89a71a8cd2d1fad6bc2a5c9 WatchSource:0}: Error finding container 0564e0d30507010d77c27b370595114e0d9f86e0f89a71a8cd2d1fad6bc2a5c9: Status 404 returned error can't find the container with id 0564e0d30507010d77c27b370595114e0d9f86e0f89a71a8cd2d1fad6bc2a5c9 Jul 07 00:03:04.680010 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.679985 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93feb81c-29f0-48d8-8001-f00293c545dc-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5\" (UID: \"93feb81c-29f0-48d8-8001-f00293c545dc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" Jul 07 00:03:04.680108 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.680049 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93feb81c-29f0-48d8-8001-f00293c545dc-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5\" (UID: \"93feb81c-29f0-48d8-8001-f00293c545dc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" Jul 07 00:03:04.680108 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.680083 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l99hd\" (UniqueName: \"kubernetes.io/projected/93feb81c-29f0-48d8-8001-f00293c545dc-kube-api-access-l99hd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5\" (UID: \"93feb81c-29f0-48d8-8001-f00293c545dc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" Jul 07 00:03:04.680348 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.680332 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93feb81c-29f0-48d8-8001-f00293c545dc-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5\" (UID: \"93feb81c-29f0-48d8-8001-f00293c545dc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" Jul 07 00:03:04.680384 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.680359 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93feb81c-29f0-48d8-8001-f00293c545dc-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5\" (UID: \"93feb81c-29f0-48d8-8001-f00293c545dc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" Jul 07 00:03:04.692323 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.692299 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l99hd\" (UniqueName: \"kubernetes.io/projected/93feb81c-29f0-48d8-8001-f00293c545dc-kube-api-access-l99hd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5\" (UID: \"93feb81c-29f0-48d8-8001-f00293c545dc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" Jul 07 00:03:04.777441 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.777374 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n"] Jul 07 00:03:04.779607 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.779592 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" Jul 07 00:03:04.793386 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.793368 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" Jul 07 00:03:04.796687 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.796667 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n"] Jul 07 00:03:04.881060 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.881030 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sptq\" (UniqueName: \"kubernetes.io/projected/0582860a-f733-4781-a75b-17f86eb6447c-kube-api-access-6sptq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n\" (UID: \"0582860a-f733-4781-a75b-17f86eb6447c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" Jul 07 00:03:04.881189 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.881074 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0582860a-f733-4781-a75b-17f86eb6447c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n\" (UID: \"0582860a-f733-4781-a75b-17f86eb6447c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" Jul 07 00:03:04.881254 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.881225 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0582860a-f733-4781-a75b-17f86eb6447c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n\" (UID: \"0582860a-f733-4781-a75b-17f86eb6447c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" Jul 07 00:03:04.918668 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.918645 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5"] Jul 07 00:03:04.920320 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:03:04.920293 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93feb81c_29f0_48d8_8001_f00293c545dc.slice/crio-dd11a3afb00a644a8e26ccff8d6898f237c8bb7b79e5d21c4f2ee9defc0a517b WatchSource:0}: Error finding container dd11a3afb00a644a8e26ccff8d6898f237c8bb7b79e5d21c4f2ee9defc0a517b: Status 404 returned error can't find the container with id dd11a3afb00a644a8e26ccff8d6898f237c8bb7b79e5d21c4f2ee9defc0a517b Jul 07 00:03:04.982598 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.982567 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0582860a-f733-4781-a75b-17f86eb6447c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n\" (UID: \"0582860a-f733-4781-a75b-17f86eb6447c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" Jul 07 00:03:04.982922 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.982655 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0582860a-f733-4781-a75b-17f86eb6447c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n\" (UID: \"0582860a-f733-4781-a75b-17f86eb6447c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" Jul 07 00:03:04.982922 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.982720 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sptq\" (UniqueName: \"kubernetes.io/projected/0582860a-f733-4781-a75b-17f86eb6447c-kube-api-access-6sptq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n\" (UID: \"0582860a-f733-4781-a75b-17f86eb6447c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" Jul 07 00:03:04.983000 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.982922 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0582860a-f733-4781-a75b-17f86eb6447c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n\" (UID: \"0582860a-f733-4781-a75b-17f86eb6447c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" Jul 07 00:03:04.983000 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.982979 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0582860a-f733-4781-a75b-17f86eb6447c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n\" (UID: \"0582860a-f733-4781-a75b-17f86eb6447c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" Jul 07 00:03:04.991686 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:04.991663 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sptq\" (UniqueName: \"kubernetes.io/projected/0582860a-f733-4781-a75b-17f86eb6447c-kube-api-access-6sptq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n\" (UID: \"0582860a-f733-4781-a75b-17f86eb6447c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" Jul 07 00:03:05.088478 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:05.088374 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" Jul 07 00:03:05.218725 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:05.218703 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n"] Jul 07 00:03:05.220266 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:03:05.220237 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0582860a_f733_4781_a75b_17f86eb6447c.slice/crio-956d86dbfe7b3793e963140d332ad929b8cda17e094f32c3db3e210dfc3445cc WatchSource:0}: Error finding container 956d86dbfe7b3793e963140d332ad929b8cda17e094f32c3db3e210dfc3445cc: Status 404 returned error can't find the container with id 956d86dbfe7b3793e963140d332ad929b8cda17e094f32c3db3e210dfc3445cc Jul 07 00:03:05.416296 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:05.416199 2568 generic.go:356] "Generic (PLEG): container finished" podID="93feb81c-29f0-48d8-8001-f00293c545dc" containerID="bf34cbda7d420b54eea697dd528922e7527fab1e79cd4c9657001dfa78f24bc4" exitCode=0 Jul 07 00:03:05.416485 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:05.416284 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" event={"ID":"93feb81c-29f0-48d8-8001-f00293c545dc","Type":"ContainerDied","Data":"bf34cbda7d420b54eea697dd528922e7527fab1e79cd4c9657001dfa78f24bc4"} Jul 07 00:03:05.416485 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:05.416381 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" event={"ID":"93feb81c-29f0-48d8-8001-f00293c545dc","Type":"ContainerStarted","Data":"dd11a3afb00a644a8e26ccff8d6898f237c8bb7b79e5d21c4f2ee9defc0a517b"} Jul 07 00:03:05.417935 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:05.417915 2568 generic.go:356] "Generic (PLEG): container finished" podID="0582860a-f733-4781-a75b-17f86eb6447c" containerID="ae748e3470bec16ae7463d67f90fe9c55167626b3f6ff003e5cea8a12eb52138" exitCode=0 Jul 07 00:03:05.418038 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:05.417982 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" event={"ID":"0582860a-f733-4781-a75b-17f86eb6447c","Type":"ContainerDied","Data":"ae748e3470bec16ae7463d67f90fe9c55167626b3f6ff003e5cea8a12eb52138"} Jul 07 00:03:05.418038 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:05.418016 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" event={"ID":"0582860a-f733-4781-a75b-17f86eb6447c","Type":"ContainerStarted","Data":"956d86dbfe7b3793e963140d332ad929b8cda17e094f32c3db3e210dfc3445cc"} Jul 07 00:03:05.419635 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:05.419522 2568 generic.go:356] "Generic (PLEG): container finished" podID="c1a71ff1-2865-4800-8b89-8c62a56ee213" containerID="cbde404e81364d8217ffe0ac7e7f247e45a91f4d7c891017536d863df5c24e5d" exitCode=0 Jul 07 00:03:05.419635 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:05.419602 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" event={"ID":"c1a71ff1-2865-4800-8b89-8c62a56ee213","Type":"ContainerDied","Data":"cbde404e81364d8217ffe0ac7e7f247e45a91f4d7c891017536d863df5c24e5d"} Jul 07 00:03:05.419741 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:05.419659 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" event={"ID":"c1a71ff1-2865-4800-8b89-8c62a56ee213","Type":"ContainerStarted","Data":"0564e0d30507010d77c27b370595114e0d9f86e0f89a71a8cd2d1fad6bc2a5c9"} Jul 07 00:03:05.421282 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:05.421263 2568 generic.go:356] "Generic (PLEG): container finished" podID="9872f2e0-1bdc-41d4-bd52-250eb055a764" containerID="e36301b1cd2170e2891c10c073201c4029bda68434d4fae8d5da5de9b8edde63" exitCode=0 Jul 07 00:03:05.421361 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:05.421287 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" event={"ID":"9872f2e0-1bdc-41d4-bd52-250eb055a764","Type":"ContainerDied","Data":"e36301b1cd2170e2891c10c073201c4029bda68434d4fae8d5da5de9b8edde63"} Jul 07 00:03:05.421361 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:05.421321 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" event={"ID":"9872f2e0-1bdc-41d4-bd52-250eb055a764","Type":"ContainerStarted","Data":"2c02cd59fc42a02a1ac09a1cd9fc709118cedcdf007199f1664b865715fee3af"} Jul 07 00:03:06.427732 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:06.427706 2568 generic.go:356] "Generic (PLEG): container finished" podID="0582860a-f733-4781-a75b-17f86eb6447c" containerID="c4f3007274c3ad8d3d1da98ef1ff183b8be2974a7d5ebd05e211f2aae08d7d71" exitCode=0 Jul 07 00:03:06.428081 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:06.427784 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" event={"ID":"0582860a-f733-4781-a75b-17f86eb6447c","Type":"ContainerDied","Data":"c4f3007274c3ad8d3d1da98ef1ff183b8be2974a7d5ebd05e211f2aae08d7d71"} Jul 07 00:03:06.429715 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:06.429696 2568 generic.go:356] "Generic (PLEG): container finished" podID="c1a71ff1-2865-4800-8b89-8c62a56ee213" containerID="031a706c6f6538e40df48bd2dde3e91030989b3d71ac2257511399926f9b2a0f" exitCode=0 Jul 07 00:03:06.429806 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:06.429777 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" event={"ID":"c1a71ff1-2865-4800-8b89-8c62a56ee213","Type":"ContainerDied","Data":"031a706c6f6538e40df48bd2dde3e91030989b3d71ac2257511399926f9b2a0f"} Jul 07 00:03:06.431905 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:06.431878 2568 generic.go:356] "Generic (PLEG): container finished" podID="9872f2e0-1bdc-41d4-bd52-250eb055a764" containerID="e246af5afe68a455c9cf4d0f27e90d7c105d813a9e288d525fbd7d72bf56cb3a" exitCode=0 Jul 07 00:03:06.432011 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:06.431963 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" event={"ID":"9872f2e0-1bdc-41d4-bd52-250eb055a764","Type":"ContainerDied","Data":"e246af5afe68a455c9cf4d0f27e90d7c105d813a9e288d525fbd7d72bf56cb3a"} Jul 07 00:03:07.438814 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:07.438783 2568 generic.go:356] "Generic (PLEG): container finished" podID="9872f2e0-1bdc-41d4-bd52-250eb055a764" containerID="6c1fcd85985211ffe6cfad3c16292d956f4a0b5a52f32da00a533ae6644438c5" exitCode=0 Jul 07 00:03:07.439182 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:07.438854 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" event={"ID":"9872f2e0-1bdc-41d4-bd52-250eb055a764","Type":"ContainerDied","Data":"6c1fcd85985211ffe6cfad3c16292d956f4a0b5a52f32da00a533ae6644438c5"} Jul 07 00:03:07.440566 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:07.440544 2568 generic.go:356] "Generic (PLEG): container finished" podID="93feb81c-29f0-48d8-8001-f00293c545dc" containerID="2ce858130cad3efabf3698ef8baa1c15efc8a1b9b720c14ce8123adab6cd31d3" exitCode=0 Jul 07 00:03:07.440688 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:07.440620 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" event={"ID":"93feb81c-29f0-48d8-8001-f00293c545dc","Type":"ContainerDied","Data":"2ce858130cad3efabf3698ef8baa1c15efc8a1b9b720c14ce8123adab6cd31d3"} Jul 07 00:03:07.442848 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:07.442826 2568 generic.go:356] "Generic (PLEG): container finished" podID="0582860a-f733-4781-a75b-17f86eb6447c" containerID="51056d045b79ec77311d988ba02ce342a1a349eebb240afeccb973da5b05421e" exitCode=0 Jul 07 00:03:07.442946 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:07.442903 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" event={"ID":"0582860a-f733-4781-a75b-17f86eb6447c","Type":"ContainerDied","Data":"51056d045b79ec77311d988ba02ce342a1a349eebb240afeccb973da5b05421e"} Jul 07 00:03:07.444898 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:07.444880 2568 generic.go:356] "Generic (PLEG): container finished" podID="c1a71ff1-2865-4800-8b89-8c62a56ee213" containerID="4bad35efb4e21bb2d7f504e704aaaa831aa64fd5e10c6ecf88c4b87e00cde80d" exitCode=0 Jul 07 00:03:07.444980 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:07.444918 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" event={"ID":"c1a71ff1-2865-4800-8b89-8c62a56ee213","Type":"ContainerDied","Data":"4bad35efb4e21bb2d7f504e704aaaa831aa64fd5e10c6ecf88c4b87e00cde80d"} Jul 07 00:03:08.453757 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.453718 2568 generic.go:356] "Generic (PLEG): container finished" podID="93feb81c-29f0-48d8-8001-f00293c545dc" containerID="5ae9fa6699b2f14cd0aa91e9607c3da9ae1c3e012056130798d277d77d92096f" exitCode=0 Jul 07 00:03:08.454239 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.453800 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" event={"ID":"93feb81c-29f0-48d8-8001-f00293c545dc","Type":"ContainerDied","Data":"5ae9fa6699b2f14cd0aa91e9607c3da9ae1c3e012056130798d277d77d92096f"} Jul 07 00:03:08.600826 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.600805 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" Jul 07 00:03:08.612794 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.612771 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9872f2e0-1bdc-41d4-bd52-250eb055a764-bundle\") pod \"9872f2e0-1bdc-41d4-bd52-250eb055a764\" (UID: \"9872f2e0-1bdc-41d4-bd52-250eb055a764\") " Jul 07 00:03:08.612924 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.612823 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqk8h\" (UniqueName: \"kubernetes.io/projected/9872f2e0-1bdc-41d4-bd52-250eb055a764-kube-api-access-zqk8h\") pod \"9872f2e0-1bdc-41d4-bd52-250eb055a764\" (UID: \"9872f2e0-1bdc-41d4-bd52-250eb055a764\") " Jul 07 00:03:08.612924 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.612913 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9872f2e0-1bdc-41d4-bd52-250eb055a764-util\") pod \"9872f2e0-1bdc-41d4-bd52-250eb055a764\" (UID: \"9872f2e0-1bdc-41d4-bd52-250eb055a764\") " Jul 07 00:03:08.613411 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.613376 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9872f2e0-1bdc-41d4-bd52-250eb055a764-bundle" (OuterVolumeSpecName: "bundle") pod "9872f2e0-1bdc-41d4-bd52-250eb055a764" (UID: "9872f2e0-1bdc-41d4-bd52-250eb055a764"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:03:08.615560 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.615532 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9872f2e0-1bdc-41d4-bd52-250eb055a764-kube-api-access-zqk8h" (OuterVolumeSpecName: "kube-api-access-zqk8h") pod "9872f2e0-1bdc-41d4-bd52-250eb055a764" (UID: "9872f2e0-1bdc-41d4-bd52-250eb055a764"). InnerVolumeSpecName "kube-api-access-zqk8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:03:08.622190 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.622159 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9872f2e0-1bdc-41d4-bd52-250eb055a764-util" (OuterVolumeSpecName: "util") pod "9872f2e0-1bdc-41d4-bd52-250eb055a764" (UID: "9872f2e0-1bdc-41d4-bd52-250eb055a764"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:03:08.644135 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.644114 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" Jul 07 00:03:08.646757 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.646740 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" Jul 07 00:03:08.713471 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.713395 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmxgv\" (UniqueName: \"kubernetes.io/projected/c1a71ff1-2865-4800-8b89-8c62a56ee213-kube-api-access-wmxgv\") pod \"c1a71ff1-2865-4800-8b89-8c62a56ee213\" (UID: \"c1a71ff1-2865-4800-8b89-8c62a56ee213\") " Jul 07 00:03:08.713471 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.713451 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1a71ff1-2865-4800-8b89-8c62a56ee213-bundle\") pod \"c1a71ff1-2865-4800-8b89-8c62a56ee213\" (UID: \"c1a71ff1-2865-4800-8b89-8c62a56ee213\") " Jul 07 00:03:08.713471 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.713473 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0582860a-f733-4781-a75b-17f86eb6447c-bundle\") pod \"0582860a-f733-4781-a75b-17f86eb6447c\" (UID: \"0582860a-f733-4781-a75b-17f86eb6447c\") " Jul 07 00:03:08.713716 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.713494 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sptq\" (UniqueName: \"kubernetes.io/projected/0582860a-f733-4781-a75b-17f86eb6447c-kube-api-access-6sptq\") pod \"0582860a-f733-4781-a75b-17f86eb6447c\" (UID: \"0582860a-f733-4781-a75b-17f86eb6447c\") " Jul 07 00:03:08.713716 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.713540 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0582860a-f733-4781-a75b-17f86eb6447c-util\") pod \"0582860a-f733-4781-a75b-17f86eb6447c\" (UID: \"0582860a-f733-4781-a75b-17f86eb6447c\") " Jul 07 00:03:08.713716 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.713576 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1a71ff1-2865-4800-8b89-8c62a56ee213-util\") pod \"c1a71ff1-2865-4800-8b89-8c62a56ee213\" (UID: \"c1a71ff1-2865-4800-8b89-8c62a56ee213\") " Jul 07 00:03:08.713861 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.713786 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9872f2e0-1bdc-41d4-bd52-250eb055a764-util\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:08.713861 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.713808 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9872f2e0-1bdc-41d4-bd52-250eb055a764-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:08.713861 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.713825 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqk8h\" (UniqueName: \"kubernetes.io/projected/9872f2e0-1bdc-41d4-bd52-250eb055a764-kube-api-access-zqk8h\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:08.714071 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.714025 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a71ff1-2865-4800-8b89-8c62a56ee213-bundle" (OuterVolumeSpecName: "bundle") pod "c1a71ff1-2865-4800-8b89-8c62a56ee213" (UID: "c1a71ff1-2865-4800-8b89-8c62a56ee213"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:03:08.714276 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.714251 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0582860a-f733-4781-a75b-17f86eb6447c-bundle" (OuterVolumeSpecName: "bundle") pod "0582860a-f733-4781-a75b-17f86eb6447c" (UID: "0582860a-f733-4781-a75b-17f86eb6447c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:03:08.715567 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.715547 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a71ff1-2865-4800-8b89-8c62a56ee213-kube-api-access-wmxgv" (OuterVolumeSpecName: "kube-api-access-wmxgv") pod "c1a71ff1-2865-4800-8b89-8c62a56ee213" (UID: "c1a71ff1-2865-4800-8b89-8c62a56ee213"). InnerVolumeSpecName "kube-api-access-wmxgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:03:08.715959 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.715944 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0582860a-f733-4781-a75b-17f86eb6447c-kube-api-access-6sptq" (OuterVolumeSpecName: "kube-api-access-6sptq") pod "0582860a-f733-4781-a75b-17f86eb6447c" (UID: "0582860a-f733-4781-a75b-17f86eb6447c"). InnerVolumeSpecName "kube-api-access-6sptq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:03:08.719908 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.719885 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0582860a-f733-4781-a75b-17f86eb6447c-util" (OuterVolumeSpecName: "util") pod "0582860a-f733-4781-a75b-17f86eb6447c" (UID: "0582860a-f733-4781-a75b-17f86eb6447c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:03:08.719957 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.719927 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a71ff1-2865-4800-8b89-8c62a56ee213-util" (OuterVolumeSpecName: "util") pod "c1a71ff1-2865-4800-8b89-8c62a56ee213" (UID: "c1a71ff1-2865-4800-8b89-8c62a56ee213"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:03:08.814101 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.814079 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0582860a-f733-4781-a75b-17f86eb6447c-util\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:08.814101 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.814098 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1a71ff1-2865-4800-8b89-8c62a56ee213-util\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:08.814205 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.814107 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmxgv\" (UniqueName: \"kubernetes.io/projected/c1a71ff1-2865-4800-8b89-8c62a56ee213-kube-api-access-wmxgv\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:08.814205 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.814116 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1a71ff1-2865-4800-8b89-8c62a56ee213-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:08.814205 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.814124 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0582860a-f733-4781-a75b-17f86eb6447c-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:08.814205 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:08.814131 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sptq\" (UniqueName: \"kubernetes.io/projected/0582860a-f733-4781-a75b-17f86eb6447c-kube-api-access-6sptq\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:09.461202 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.461169 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" event={"ID":"c1a71ff1-2865-4800-8b89-8c62a56ee213","Type":"ContainerDied","Data":"0564e0d30507010d77c27b370595114e0d9f86e0f89a71a8cd2d1fad6bc2a5c9"} Jul 07 00:03:09.461630 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.461217 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0564e0d30507010d77c27b370595114e0d9f86e0f89a71a8cd2d1fad6bc2a5c9" Jul 07 00:03:09.461630 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.461176 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl" Jul 07 00:03:09.463240 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.463217 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" Jul 07 00:03:09.463356 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.463250 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl" event={"ID":"9872f2e0-1bdc-41d4-bd52-250eb055a764","Type":"ContainerDied","Data":"2c02cd59fc42a02a1ac09a1cd9fc709118cedcdf007199f1664b865715fee3af"} Jul 07 00:03:09.463356 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.463298 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c02cd59fc42a02a1ac09a1cd9fc709118cedcdf007199f1664b865715fee3af" Jul 07 00:03:09.465636 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.465614 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" Jul 07 00:03:09.465636 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.465626 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n" event={"ID":"0582860a-f733-4781-a75b-17f86eb6447c","Type":"ContainerDied","Data":"956d86dbfe7b3793e963140d332ad929b8cda17e094f32c3db3e210dfc3445cc"} Jul 07 00:03:09.465776 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.465673 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="956d86dbfe7b3793e963140d332ad929b8cda17e094f32c3db3e210dfc3445cc" Jul 07 00:03:09.595362 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.595342 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" Jul 07 00:03:09.620683 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.620655 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l99hd\" (UniqueName: \"kubernetes.io/projected/93feb81c-29f0-48d8-8001-f00293c545dc-kube-api-access-l99hd\") pod \"93feb81c-29f0-48d8-8001-f00293c545dc\" (UID: \"93feb81c-29f0-48d8-8001-f00293c545dc\") " Jul 07 00:03:09.620776 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.620718 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93feb81c-29f0-48d8-8001-f00293c545dc-util\") pod \"93feb81c-29f0-48d8-8001-f00293c545dc\" (UID: \"93feb81c-29f0-48d8-8001-f00293c545dc\") " Jul 07 00:03:09.620776 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.620754 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93feb81c-29f0-48d8-8001-f00293c545dc-bundle\") pod \"93feb81c-29f0-48d8-8001-f00293c545dc\" (UID: \"93feb81c-29f0-48d8-8001-f00293c545dc\") " Jul 07 00:03:09.621182 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.621158 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93feb81c-29f0-48d8-8001-f00293c545dc-bundle" (OuterVolumeSpecName: "bundle") pod "93feb81c-29f0-48d8-8001-f00293c545dc" (UID: "93feb81c-29f0-48d8-8001-f00293c545dc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:03:09.622805 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.622781 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93feb81c-29f0-48d8-8001-f00293c545dc-kube-api-access-l99hd" (OuterVolumeSpecName: "kube-api-access-l99hd") pod "93feb81c-29f0-48d8-8001-f00293c545dc" (UID: "93feb81c-29f0-48d8-8001-f00293c545dc"). InnerVolumeSpecName "kube-api-access-l99hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:03:09.628259 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.628240 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93feb81c-29f0-48d8-8001-f00293c545dc-util" (OuterVolumeSpecName: "util") pod "93feb81c-29f0-48d8-8001-f00293c545dc" (UID: "93feb81c-29f0-48d8-8001-f00293c545dc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:03:09.721698 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.721638 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93feb81c-29f0-48d8-8001-f00293c545dc-util\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:09.721698 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.721659 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93feb81c-29f0-48d8-8001-f00293c545dc-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:09.721698 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:09.721669 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l99hd\" (UniqueName: \"kubernetes.io/projected/93feb81c-29f0-48d8-8001-f00293c545dc-kube-api-access-l99hd\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:10.473854 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:10.473817 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" Jul 07 00:03:10.473854 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:10.473837 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5" event={"ID":"93feb81c-29f0-48d8-8001-f00293c545dc","Type":"ContainerDied","Data":"dd11a3afb00a644a8e26ccff8d6898f237c8bb7b79e5d21c4f2ee9defc0a517b"} Jul 07 00:03:10.474467 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:10.473888 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd11a3afb00a644a8e26ccff8d6898f237c8bb7b79e5d21c4f2ee9defc0a517b" Jul 07 00:03:16.646505 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646478 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d8b5fc77c-4bxrq"] Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646736 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a71ff1-2865-4800-8b89-8c62a56ee213" containerName="pull" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646746 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a71ff1-2865-4800-8b89-8c62a56ee213" containerName="pull" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646758 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0582860a-f733-4781-a75b-17f86eb6447c" containerName="extract" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646764 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0582860a-f733-4781-a75b-17f86eb6447c" containerName="extract" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646772 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93feb81c-29f0-48d8-8001-f00293c545dc" containerName="extract" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646777 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="93feb81c-29f0-48d8-8001-f00293c545dc" containerName="extract" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646789 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a71ff1-2865-4800-8b89-8c62a56ee213" containerName="extract" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646794 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a71ff1-2865-4800-8b89-8c62a56ee213" containerName="extract" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646802 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0582860a-f733-4781-a75b-17f86eb6447c" containerName="util" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646807 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0582860a-f733-4781-a75b-17f86eb6447c" containerName="util" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646814 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93feb81c-29f0-48d8-8001-f00293c545dc" containerName="util" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646819 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="93feb81c-29f0-48d8-8001-f00293c545dc" containerName="util" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646825 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93feb81c-29f0-48d8-8001-f00293c545dc" containerName="pull" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646830 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="93feb81c-29f0-48d8-8001-f00293c545dc" containerName="pull" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646839 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9872f2e0-1bdc-41d4-bd52-250eb055a764" containerName="extract" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646844 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9872f2e0-1bdc-41d4-bd52-250eb055a764" containerName="extract" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646850 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a71ff1-2865-4800-8b89-8c62a56ee213" containerName="util" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646854 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a71ff1-2865-4800-8b89-8c62a56ee213" containerName="util" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646859 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9872f2e0-1bdc-41d4-bd52-250eb055a764" containerName="pull" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646864 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9872f2e0-1bdc-41d4-bd52-250eb055a764" containerName="pull" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646870 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9872f2e0-1bdc-41d4-bd52-250eb055a764" containerName="util" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646875 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9872f2e0-1bdc-41d4-bd52-250eb055a764" containerName="util" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646880 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0582860a-f733-4781-a75b-17f86eb6447c" containerName="pull" Jul 07 00:03:16.646874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646885 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0582860a-f733-4781-a75b-17f86eb6447c" containerName="pull" Jul 07 00:03:16.647561 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646930 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0582860a-f733-4781-a75b-17f86eb6447c" containerName="extract" Jul 07 00:03:16.647561 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646937 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="93feb81c-29f0-48d8-8001-f00293c545dc" containerName="extract" Jul 07 00:03:16.647561 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646943 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a71ff1-2865-4800-8b89-8c62a56ee213" containerName="extract" Jul 07 00:03:16.647561 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.646947 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9872f2e0-1bdc-41d4-bd52-250eb055a764" containerName="extract" Jul 07 00:03:16.649774 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.649759 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.673614 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.673591 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/361d91ec-77f2-4f7f-b082-f0718ebf420c-service-ca\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.673720 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.673620 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361d91ec-77f2-4f7f-b082-f0718ebf420c-trusted-ca-bundle\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.673720 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.673658 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/361d91ec-77f2-4f7f-b082-f0718ebf420c-oauth-serving-cert\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.673720 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.673679 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/361d91ec-77f2-4f7f-b082-f0718ebf420c-console-config\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.673720 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.673697 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xcv\" (UniqueName: \"kubernetes.io/projected/361d91ec-77f2-4f7f-b082-f0718ebf420c-kube-api-access-z8xcv\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.673868 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.673747 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/361d91ec-77f2-4f7f-b082-f0718ebf420c-console-serving-cert\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.673868 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.673766 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/361d91ec-77f2-4f7f-b082-f0718ebf420c-console-oauth-config\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.774450 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.774408 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/361d91ec-77f2-4f7f-b082-f0718ebf420c-console-serving-cert\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.774556 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.774454 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/361d91ec-77f2-4f7f-b082-f0718ebf420c-console-oauth-config\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.774556 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.774475 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/361d91ec-77f2-4f7f-b082-f0718ebf420c-service-ca\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.774556 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.774497 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361d91ec-77f2-4f7f-b082-f0718ebf420c-trusted-ca-bundle\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.774556 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.774517 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/361d91ec-77f2-4f7f-b082-f0718ebf420c-oauth-serving-cert\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.774556 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.774541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/361d91ec-77f2-4f7f-b082-f0718ebf420c-console-config\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.774784 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.774569 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xcv\" (UniqueName: \"kubernetes.io/projected/361d91ec-77f2-4f7f-b082-f0718ebf420c-kube-api-access-z8xcv\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.775386 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.775333 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/361d91ec-77f2-4f7f-b082-f0718ebf420c-oauth-serving-cert\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.775520 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.775383 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361d91ec-77f2-4f7f-b082-f0718ebf420c-trusted-ca-bundle\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.775520 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.775412 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/361d91ec-77f2-4f7f-b082-f0718ebf420c-service-ca\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.775520 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.775501 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/361d91ec-77f2-4f7f-b082-f0718ebf420c-console-config\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.777629 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.777598 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/361d91ec-77f2-4f7f-b082-f0718ebf420c-console-oauth-config\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.777814 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.777764 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/361d91ec-77f2-4f7f-b082-f0718ebf420c-console-serving-cert\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.779447 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.779409 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d8b5fc77c-4bxrq"] Jul 07 00:03:16.793139 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.793112 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xcv\" (UniqueName: \"kubernetes.io/projected/361d91ec-77f2-4f7f-b082-f0718ebf420c-kube-api-access-z8xcv\") pod \"console-6d8b5fc77c-4bxrq\" (UID: \"361d91ec-77f2-4f7f-b082-f0718ebf420c\") " pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:16.958630 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:16.958602 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:17.118144 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:17.118116 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d8b5fc77c-4bxrq"] Jul 07 00:03:17.120094 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:03:17.120068 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361d91ec_77f2_4f7f_b082_f0718ebf420c.slice/crio-b331308194643cf6c386049eb5e6064c4d0a1321ff7b1a2b5e7995c6c920d776 WatchSource:0}: Error finding container b331308194643cf6c386049eb5e6064c4d0a1321ff7b1a2b5e7995c6c920d776: Status 404 returned error can't find the container with id b331308194643cf6c386049eb5e6064c4d0a1321ff7b1a2b5e7995c6c920d776 Jul 07 00:03:17.506083 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:17.506048 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8b5fc77c-4bxrq" event={"ID":"361d91ec-77f2-4f7f-b082-f0718ebf420c","Type":"ContainerStarted","Data":"791d20a13d9db8600c541ae1d3c68396e2f42594e53e0e688b48ab1b61189dd2"} Jul 07 00:03:17.506275 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:17.506099 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8b5fc77c-4bxrq" event={"ID":"361d91ec-77f2-4f7f-b082-f0718ebf420c","Type":"ContainerStarted","Data":"b331308194643cf6c386049eb5e6064c4d0a1321ff7b1a2b5e7995c6c920d776"} Jul 07 00:03:17.560893 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:17.560849 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d8b5fc77c-4bxrq" podStartSLOduration=1.5608338640000001 podStartE2EDuration="1.560833864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:03:17.559226895 +0000 UTC m=+420.699683951" watchObservedRunningTime="2026-07-07 00:03:17.560833864 +0000 UTC m=+420.701290920" Jul 07 00:03:26.959723 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:26.959686 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:26.960173 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:26.959744 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:26.964374 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:26.964353 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:27.555077 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:27.555046 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d8b5fc77c-4bxrq" Jul 07 00:03:27.636957 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:27.636929 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54d54cffc5-wdhl4"] Jul 07 00:03:52.660138 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:52.660050 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-54d54cffc5-wdhl4" podUID="c5a75e2f-3431-4ba4-9e25-e560f629a67c" containerName="console" containerID="cri-o://56c8daf2c5ac1f5aee81b660a0908b1931f368ac491c5ce23ed69f28252c2390" gracePeriod=15 Jul 07 00:03:52.904012 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:52.903985 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d54cffc5-wdhl4" Jul 07 00:03:53.050265 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.050226 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-oauth-config\") pod \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " Jul 07 00:03:53.050487 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.050297 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-service-ca\") pod \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " Jul 07 00:03:53.050487 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.050319 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-oauth-serving-cert\") pod \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " Jul 07 00:03:53.050487 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.050353 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr424\" (UniqueName: \"kubernetes.io/projected/c5a75e2f-3431-4ba4-9e25-e560f629a67c-kube-api-access-zr424\") pod \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " Jul 07 00:03:53.050487 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.050395 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-trusted-ca-bundle\") pod \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " Jul 07 00:03:53.050487 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.050447 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-serving-cert\") pod \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " Jul 07 00:03:53.050766 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.050491 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-config\") pod \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\" (UID: \"c5a75e2f-3431-4ba4-9e25-e560f629a67c\") " Jul 07 00:03:53.050955 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.050849 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-service-ca" (OuterVolumeSpecName: "service-ca") pod "c5a75e2f-3431-4ba4-9e25-e560f629a67c" (UID: "c5a75e2f-3431-4ba4-9e25-e560f629a67c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:03:53.050955 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.050855 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c5a75e2f-3431-4ba4-9e25-e560f629a67c" (UID: "c5a75e2f-3431-4ba4-9e25-e560f629a67c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:03:53.050955 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.050874 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-config" (OuterVolumeSpecName: "console-config") pod "c5a75e2f-3431-4ba4-9e25-e560f629a67c" (UID: "c5a75e2f-3431-4ba4-9e25-e560f629a67c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:03:53.050955 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.050893 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c5a75e2f-3431-4ba4-9e25-e560f629a67c" (UID: "c5a75e2f-3431-4ba4-9e25-e560f629a67c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:03:53.052565 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.052535 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c5a75e2f-3431-4ba4-9e25-e560f629a67c" (UID: "c5a75e2f-3431-4ba4-9e25-e560f629a67c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:03:53.052667 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.052635 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c5a75e2f-3431-4ba4-9e25-e560f629a67c" (UID: "c5a75e2f-3431-4ba4-9e25-e560f629a67c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:03:53.052715 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.052702 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a75e2f-3431-4ba4-9e25-e560f629a67c-kube-api-access-zr424" (OuterVolumeSpecName: "kube-api-access-zr424") pod "c5a75e2f-3431-4ba4-9e25-e560f629a67c" (UID: "c5a75e2f-3431-4ba4-9e25-e560f629a67c"). InnerVolumeSpecName "kube-api-access-zr424". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:03:53.151277 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.151239 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr424\" (UniqueName: \"kubernetes.io/projected/c5a75e2f-3431-4ba4-9e25-e560f629a67c-kube-api-access-zr424\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:53.151277 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.151270 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-trusted-ca-bundle\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:53.151277 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.151280 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-serving-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:53.151556 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.151290 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-config\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:53.151556 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.151301 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a75e2f-3431-4ba4-9e25-e560f629a67c-console-oauth-config\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:53.151556 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.151309 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:53.151556 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.151317 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a75e2f-3431-4ba4-9e25-e560f629a67c-oauth-serving-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:03:53.673275 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.673240 2568 generic.go:356] "Generic (PLEG): container finished" podID="c5a75e2f-3431-4ba4-9e25-e560f629a67c" containerID="56c8daf2c5ac1f5aee81b660a0908b1931f368ac491c5ce23ed69f28252c2390" exitCode=0 Jul 07 00:03:53.673739 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.673283 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d54cffc5-wdhl4" Jul 07 00:03:53.673739 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.673327 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d54cffc5-wdhl4" event={"ID":"c5a75e2f-3431-4ba4-9e25-e560f629a67c","Type":"ContainerDied","Data":"56c8daf2c5ac1f5aee81b660a0908b1931f368ac491c5ce23ed69f28252c2390"} Jul 07 00:03:53.673739 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.673385 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d54cffc5-wdhl4" event={"ID":"c5a75e2f-3431-4ba4-9e25-e560f629a67c","Type":"ContainerDied","Data":"17a81d1f54502a8339d4fb4389086acc6793ec07badb405eae103d3d6a218f08"} Jul 07 00:03:53.673739 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.673415 2568 scope.go:117] "RemoveContainer" containerID="56c8daf2c5ac1f5aee81b660a0908b1931f368ac491c5ce23ed69f28252c2390" Jul 07 00:03:53.681745 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.681730 2568 scope.go:117] "RemoveContainer" containerID="56c8daf2c5ac1f5aee81b660a0908b1931f368ac491c5ce23ed69f28252c2390" Jul 07 00:03:53.682023 ip-10-0-129-125 kubenswrapper[2568]: E0707 00:03:53.682004 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c8daf2c5ac1f5aee81b660a0908b1931f368ac491c5ce23ed69f28252c2390\": container with ID starting with 56c8daf2c5ac1f5aee81b660a0908b1931f368ac491c5ce23ed69f28252c2390 not found: ID does not exist" containerID="56c8daf2c5ac1f5aee81b660a0908b1931f368ac491c5ce23ed69f28252c2390" Jul 07 00:03:53.682080 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.682030 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c8daf2c5ac1f5aee81b660a0908b1931f368ac491c5ce23ed69f28252c2390"} err="failed to get container status \"56c8daf2c5ac1f5aee81b660a0908b1931f368ac491c5ce23ed69f28252c2390\": rpc error: code = NotFound desc = could not find container \"56c8daf2c5ac1f5aee81b660a0908b1931f368ac491c5ce23ed69f28252c2390\": container with ID starting with 56c8daf2c5ac1f5aee81b660a0908b1931f368ac491c5ce23ed69f28252c2390 not found: ID does not exist" Jul 07 00:03:53.691715 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.691692 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54d54cffc5-wdhl4"] Jul 07 00:03:53.699151 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:53.699131 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54d54cffc5-wdhl4"] Jul 07 00:03:55.486494 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:03:55.486464 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a75e2f-3431-4ba4-9e25-e560f629a67c" path="/var/lib/kubelet/pods/c5a75e2f-3431-4ba4-9e25-e560f629a67c/volumes" Jul 07 00:04:28.772660 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:28.772615 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:04:28.773874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:28.773843 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a75e2f-3431-4ba4-9e25-e560f629a67c" containerName="console" Jul 07 00:04:28.774040 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:28.774028 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a75e2f-3431-4ba4-9e25-e560f629a67c" containerName="console" Jul 07 00:04:28.774320 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:28.774308 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a75e2f-3431-4ba4-9e25-e560f629a67c" containerName="console" Jul 07 00:04:28.779323 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:28.779304 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-4c2k5" Jul 07 00:04:28.781730 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:28.781706 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 00:04:28.782115 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:28.782094 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:04:28.782360 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:28.782340 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 00:04:28.782360 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:28.782355 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4nstc\"" Jul 07 00:04:28.782546 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:28.782359 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 00:04:28.816456 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:28.816415 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:04:28.917454 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:28.917411 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eb2d7414-4503-4b22-ac1b-51fa9c412719-config-file\") pod \"limitador-limitador-54d6d988d5-4c2k5\" (UID: \"eb2d7414-4503-4b22-ac1b-51fa9c412719\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4c2k5" Jul 07 00:04:28.917541 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:28.917464 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nkkr\" (UniqueName: \"kubernetes.io/projected/eb2d7414-4503-4b22-ac1b-51fa9c412719-kube-api-access-4nkkr\") pod \"limitador-limitador-54d6d988d5-4c2k5\" (UID: \"eb2d7414-4503-4b22-ac1b-51fa9c412719\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4c2k5" Jul 07 00:04:29.018731 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:29.018709 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eb2d7414-4503-4b22-ac1b-51fa9c412719-config-file\") pod \"limitador-limitador-54d6d988d5-4c2k5\" (UID: \"eb2d7414-4503-4b22-ac1b-51fa9c412719\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4c2k5" Jul 07 00:04:29.018820 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:29.018741 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nkkr\" (UniqueName: \"kubernetes.io/projected/eb2d7414-4503-4b22-ac1b-51fa9c412719-kube-api-access-4nkkr\") pod \"limitador-limitador-54d6d988d5-4c2k5\" (UID: \"eb2d7414-4503-4b22-ac1b-51fa9c412719\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4c2k5" Jul 07 00:04:29.019314 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:29.019295 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eb2d7414-4503-4b22-ac1b-51fa9c412719-config-file\") pod \"limitador-limitador-54d6d988d5-4c2k5\" (UID: \"eb2d7414-4503-4b22-ac1b-51fa9c412719\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4c2k5" Jul 07 00:04:29.026814 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:29.026758 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nkkr\" (UniqueName: \"kubernetes.io/projected/eb2d7414-4503-4b22-ac1b-51fa9c412719-kube-api-access-4nkkr\") pod \"limitador-limitador-54d6d988d5-4c2k5\" (UID: \"eb2d7414-4503-4b22-ac1b-51fa9c412719\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4c2k5" Jul 07 00:04:29.090505 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:29.090466 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-4c2k5" Jul 07 00:04:29.217485 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:29.217464 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:04:29.220131 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:04:29.220104 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2d7414_4503_4b22_ac1b_51fa9c412719.slice/crio-25eafd1238cda8f583e9f1c6681d7ed46819545b3df3c0cb14d6fd292dd3bd68 WatchSource:0}: Error finding container 25eafd1238cda8f583e9f1c6681d7ed46819545b3df3c0cb14d6fd292dd3bd68: Status 404 returned error can't find the container with id 25eafd1238cda8f583e9f1c6681d7ed46819545b3df3c0cb14d6fd292dd3bd68 Jul 07 00:04:29.855624 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:29.855528 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-4c2k5" event={"ID":"eb2d7414-4503-4b22-ac1b-51fa9c412719","Type":"ContainerStarted","Data":"25eafd1238cda8f583e9f1c6681d7ed46819545b3df3c0cb14d6fd292dd3bd68"} Jul 07 00:04:31.867580 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:31.867553 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-4c2k5" event={"ID":"eb2d7414-4503-4b22-ac1b-51fa9c412719","Type":"ContainerStarted","Data":"88573469bfef8993eb69faee820705439f8a1ec10c22419af67b48e54636f4f4"} Jul 07 00:04:31.867922 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:31.867621 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-4c2k5" Jul 07 00:04:31.885620 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:31.885540 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-4c2k5" podStartSLOduration=1.47467998 podStartE2EDuration="3.885524294s" totalImagesPullingTime="2.410844314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:04:31.883693075 +0000 UTC m=+495.024150150" watchObservedRunningTime="2026-07-07 00:04:31.885524294 +0000 UTC m=+495.025981350" Jul 07 00:04:42.873890 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:04:42.873864 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-4c2k5" Jul 07 00:05:40.219228 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:05:40.219193 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:05:59.143431 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:05:59.143395 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:06:19.615994 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:19.615954 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:06:22.381338 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.381304 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq"] Jul 07 00:06:22.383592 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.383573 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.385792 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.385769 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bvnll\"" Jul 07 00:06:22.385886 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.385866 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 07 00:06:22.385964 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.385945 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 00:06:22.386006 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.385998 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 00:06:22.397043 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.397014 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq"] Jul 07 00:06:22.488906 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.488874 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6280ca7c-b127-45b1-9b6b-769069baa5f5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.488906 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.488908 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6280ca7c-b127-45b1-9b6b-769069baa5f5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.489125 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.488939 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6280ca7c-b127-45b1-9b6b-769069baa5f5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.489125 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.489004 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6280ca7c-b127-45b1-9b6b-769069baa5f5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.489125 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.489041 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6280ca7c-b127-45b1-9b6b-769069baa5f5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.489125 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.489063 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29dnw\" (UniqueName: \"kubernetes.io/projected/6280ca7c-b127-45b1-9b6b-769069baa5f5-kube-api-access-29dnw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.589357 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.589326 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6280ca7c-b127-45b1-9b6b-769069baa5f5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.589357 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.589361 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6280ca7c-b127-45b1-9b6b-769069baa5f5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.589593 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.589381 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6280ca7c-b127-45b1-9b6b-769069baa5f5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.589593 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.589465 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29dnw\" (UniqueName: \"kubernetes.io/projected/6280ca7c-b127-45b1-9b6b-769069baa5f5-kube-api-access-29dnw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.589593 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.589527 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6280ca7c-b127-45b1-9b6b-769069baa5f5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.589593 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.589585 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6280ca7c-b127-45b1-9b6b-769069baa5f5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.589946 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.589787 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6280ca7c-b127-45b1-9b6b-769069baa5f5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.589946 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.589889 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6280ca7c-b127-45b1-9b6b-769069baa5f5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.590168 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.590010 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6280ca7c-b127-45b1-9b6b-769069baa5f5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.591642 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.591619 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6280ca7c-b127-45b1-9b6b-769069baa5f5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.592032 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.592013 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6280ca7c-b127-45b1-9b6b-769069baa5f5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.597827 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.597805 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29dnw\" (UniqueName: \"kubernetes.io/projected/6280ca7c-b127-45b1-9b6b-769069baa5f5-kube-api-access-29dnw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4wkq\" (UID: \"6280ca7c-b127-45b1-9b6b-769069baa5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.693583 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.693555 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:22.820828 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:22.820806 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq"] Jul 07 00:06:22.823355 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:06:22.823323 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6280ca7c_b127_45b1_9b6b_769069baa5f5.slice/crio-e7802b2c4b54249d32bec5fab54706b870536af7463eefda1c9b5d3e2be28346 WatchSource:0}: Error finding container e7802b2c4b54249d32bec5fab54706b870536af7463eefda1c9b5d3e2be28346: Status 404 returned error can't find the container with id e7802b2c4b54249d32bec5fab54706b870536af7463eefda1c9b5d3e2be28346 Jul 07 00:06:23.398658 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:23.398616 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" event={"ID":"6280ca7c-b127-45b1-9b6b-769069baa5f5","Type":"ContainerStarted","Data":"e7802b2c4b54249d32bec5fab54706b870536af7463eefda1c9b5d3e2be28346"} Jul 07 00:06:24.013777 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:24.013739 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:06:24.405875 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:24.405838 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" event={"ID":"6280ca7c-b127-45b1-9b6b-769069baa5f5","Type":"ContainerStarted","Data":"27fcfbefcdceb8ebaddbf6498af4d4773184ad967c44b94b752b9a56829372e1"} Jul 07 00:06:24.429046 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:24.428993 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" podStartSLOduration=1.428885323 podStartE2EDuration="2.428978626s" totalImagesPullingTime="1.000093303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:06:24.426797669 +0000 UTC m=+607.567254725" watchObservedRunningTime="2026-07-07 00:06:24.428978626 +0000 UTC m=+607.569435681" Jul 07 00:06:32.694411 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:32.694369 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:32.697039 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:32.694455 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:32.707201 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:32.707180 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:32.724512 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:32.724454 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:06:33.462981 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:33.462941 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4wkq" Jul 07 00:06:40.816528 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:40.816447 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:06:49.614693 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:06:49.614652 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:08:43.831043 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:08:43.831005 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:08:53.216534 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:08:53.216495 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:09:05.414097 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:09:05.414063 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:09:14.311413 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:09:14.311384 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:09:25.818761 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:09:25.818722 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:09:50.916539 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:09:50.916462 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:10:02.513410 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:10:02.513373 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:10:36.325078 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:10:36.325037 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:10:45.637254 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:10:45.637226 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:10:54.216760 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:10:54.216714 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:11:05.724882 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:11:05.724844 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:11:15.918913 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:11:15.918818 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:11:30.923069 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:11:30.923029 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:11:37.216770 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:11:37.216733 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:11:46.613824 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:11:46.613791 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:12:08.219095 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:12:08.219061 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:12:17.118110 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:12:17.118080 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:12:41.813672 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:12:41.813590 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:12:51.919383 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:12:51.919345 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:13:07.320568 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:13:07.320536 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:13:18.723744 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:13:18.723704 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:13:28.915232 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:13:28.915192 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:13:37.503893 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:13:37.503862 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:13:46.436358 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:13:46.436319 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:15:36.117506 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:15:36.117470 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:15:47.320486 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:15:47.320387 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:15:55.917356 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:15:55.917317 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:16:07.424471 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:16:07.424437 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:16:17.328465 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:16:17.328432 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:16:27.824609 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:16:27.824576 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:16:37.315851 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:16:37.315736 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:16:47.021906 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:16:47.021869 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:16:59.210618 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:16:59.210582 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:17:07.920480 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:17:07.920453 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:17:18.713476 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:17:18.713375 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:17:43.321131 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:17:43.321097 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:17:50.616855 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:17:50.616823 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:18:00.215613 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:18:00.215577 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:18:09.320078 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:18:09.320043 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:18:16.008855 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:18:16.008821 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:18:31.919903 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:18:31.919851 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:18:43.721294 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:18:43.721212 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:18:52.714440 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:18:52.714395 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:19:03.218003 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:19:03.217968 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:19:12.317284 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:19:12.317247 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:19:21.719184 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:19:21.719141 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:19:35.020282 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:19:35.020247 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:19:44.123228 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:19:44.123196 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:19:53.817861 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:19:53.817829 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:20:03.115070 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:20:03.115034 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:20:16.122955 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:20:16.122871 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:20:24.017666 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:20:24.017632 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:20:35.724438 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:20:35.724390 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:20:48.764141 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:20:48.764108 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:21:01.817751 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:01.817720 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:21:14.619544 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:14.619510 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:21:26.520708 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:26.520674 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:21:36.708409 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:36.708371 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:21:45.100978 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.100941 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj"] Jul 07 00:21:45.104663 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.104640 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.107136 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.107115 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7a16f626-openshift-default-dockercfg-bd8tt\"" Jul 07 00:21:45.117641 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.117613 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj"] Jul 07 00:21:45.230442 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.230396 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-workload-socket\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.230634 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.230483 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istiod-ca-cert\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.230634 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.230513 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-data\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.230634 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.230573 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-podinfo\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.230634 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.230602 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4blg5\" (UniqueName: \"kubernetes.io/projected/6ea199d3-a76f-4e1e-82dd-2f794956bec0-kube-api-access-4blg5\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.230805 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.230647 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-credential-socket\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.230805 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.230663 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-token\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.230805 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.230685 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-workload-certs\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.230805 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.230731 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-envoy\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.331269 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.331227 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-envoy\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.331269 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.331271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-workload-socket\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.331562 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.331304 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istiod-ca-cert\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.331562 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.331333 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-data\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.331562 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.331378 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-podinfo\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.331562 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.331405 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4blg5\" (UniqueName: \"kubernetes.io/projected/6ea199d3-a76f-4e1e-82dd-2f794956bec0-kube-api-access-4blg5\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.331562 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.331465 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-credential-socket\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.331562 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.331489 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-token\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.331562 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.331516 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-workload-certs\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.331926 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.331740 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-workload-socket\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.331926 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.331834 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-data\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.331926 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.331891 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-credential-socket\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.332077 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.332030 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-workload-certs\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.332159 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.332141 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istiod-ca-cert\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.333766 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.333746 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-envoy\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.333854 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.333802 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-podinfo\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.339484 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.339458 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-token\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.339658 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.339640 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4blg5\" (UniqueName: \"kubernetes.io/projected/6ea199d3-a76f-4e1e-82dd-2f794956bec0-kube-api-access-4blg5\") pod \"e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.415815 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.415739 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:45.557435 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.557392 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj"] Jul 07 00:21:45.557927 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:21:45.557902 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea199d3_a76f_4e1e_82dd_2f794956bec0.slice/crio-811c004bfa73708d52bf48a79f537770e811f1d9f1efed3e2e554273c2fc5cf9 WatchSource:0}: Error finding container 811c004bfa73708d52bf48a79f537770e811f1d9f1efed3e2e554273c2fc5cf9: Status 404 returned error can't find the container with id 811c004bfa73708d52bf48a79f537770e811f1d9f1efed3e2e554273c2fc5cf9 Jul 07 00:21:45.560092 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.560059 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:21:45.560172 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.560123 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:21:45.560172 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.560149 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:21:45.631328 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.631302 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:21:45.780466 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.780409 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" event={"ID":"6ea199d3-a76f-4e1e-82dd-2f794956bec0","Type":"ContainerStarted","Data":"a16799fe2839138c72d16703287d06c40641c50870bb0c0c1d27832aa75ca20a"} Jul 07 00:21:45.780665 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.780493 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" event={"ID":"6ea199d3-a76f-4e1e-82dd-2f794956bec0","Type":"ContainerStarted","Data":"811c004bfa73708d52bf48a79f537770e811f1d9f1efed3e2e554273c2fc5cf9"} Jul 07 00:21:45.804398 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:45.804344 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" podStartSLOduration=0.804294813 podStartE2EDuration="804.294813ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:21:45.803682711 +0000 UTC m=+1528.944139770" watchObservedRunningTime="2026-07-07 00:21:45.804294813 +0000 UTC m=+1528.944751870" Jul 07 00:21:46.416543 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:46.416511 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:46.421240 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:46.421218 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:46.785104 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:46.785074 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:46.786077 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:46.786055 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:57.410476 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:57.410440 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj"] Jul 07 00:21:57.410975 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:57.410628 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" podUID="6ea199d3-a76f-4e1e-82dd-2f794956bec0" containerName="istio-proxy" containerID="cri-o://a16799fe2839138c72d16703287d06c40641c50870bb0c0c1d27832aa75ca20a" gracePeriod=30 Jul 07 00:21:58.562678 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.562652 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:58.745047 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745011 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istiod-ca-cert\") pod \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " Jul 07 00:21:58.745047 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745049 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4blg5\" (UniqueName: \"kubernetes.io/projected/6ea199d3-a76f-4e1e-82dd-2f794956bec0-kube-api-access-4blg5\") pod \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " Jul 07 00:21:58.745299 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745094 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-credential-socket\") pod \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " Jul 07 00:21:58.745299 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745127 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-workload-socket\") pod \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " Jul 07 00:21:58.745299 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745202 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-data\") pod \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " Jul 07 00:21:58.745299 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745233 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-podinfo\") pod \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " Jul 07 00:21:58.745299 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745265 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-token\") pod \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " Jul 07 00:21:58.745592 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745304 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-workload-certs\") pod \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " Jul 07 00:21:58.745592 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745343 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-envoy\") pod \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\" (UID: \"6ea199d3-a76f-4e1e-82dd-2f794956bec0\") " Jul 07 00:21:58.745592 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745346 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ea199d3-a76f-4e1e-82dd-2f794956bec0" (UID: "6ea199d3-a76f-4e1e-82dd-2f794956bec0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:58.745592 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745516 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ea199d3-a76f-4e1e-82dd-2f794956bec0" (UID: "6ea199d3-a76f-4e1e-82dd-2f794956bec0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:21:58.745592 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745558 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ea199d3-a76f-4e1e-82dd-2f794956bec0" (UID: "6ea199d3-a76f-4e1e-82dd-2f794956bec0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:58.745835 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745668 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-workload-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:21:58.745835 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745679 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ea199d3-a76f-4e1e-82dd-2f794956bec0" (UID: "6ea199d3-a76f-4e1e-82dd-2f794956bec0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:58.745835 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745693 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istiod-ca-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:21:58.745835 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.745707 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-credential-socket\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:21:58.746536 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.746509 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ea199d3-a76f-4e1e-82dd-2f794956bec0" (UID: "6ea199d3-a76f-4e1e-82dd-2f794956bec0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:58.747355 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.747331 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ea199d3-a76f-4e1e-82dd-2f794956bec0" (UID: "6ea199d3-a76f-4e1e-82dd-2f794956bec0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:21:58.747470 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.747446 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea199d3-a76f-4e1e-82dd-2f794956bec0-kube-api-access-4blg5" (OuterVolumeSpecName: "kube-api-access-4blg5") pod "6ea199d3-a76f-4e1e-82dd-2f794956bec0" (UID: "6ea199d3-a76f-4e1e-82dd-2f794956bec0"). InnerVolumeSpecName "kube-api-access-4blg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:21:58.747749 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.747726 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ea199d3-a76f-4e1e-82dd-2f794956bec0" (UID: "6ea199d3-a76f-4e1e-82dd-2f794956bec0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:21:58.747749 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.747736 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ea199d3-a76f-4e1e-82dd-2f794956bec0" (UID: "6ea199d3-a76f-4e1e-82dd-2f794956bec0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:58.842536 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.842503 2568 generic.go:356] "Generic (PLEG): container finished" podID="6ea199d3-a76f-4e1e-82dd-2f794956bec0" containerID="a16799fe2839138c72d16703287d06c40641c50870bb0c0c1d27832aa75ca20a" exitCode=0 Jul 07 00:21:58.842717 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.842548 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" Jul 07 00:21:58.842717 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.842594 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" event={"ID":"6ea199d3-a76f-4e1e-82dd-2f794956bec0","Type":"ContainerDied","Data":"a16799fe2839138c72d16703287d06c40641c50870bb0c0c1d27832aa75ca20a"} Jul 07 00:21:58.842717 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.842687 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj" event={"ID":"6ea199d3-a76f-4e1e-82dd-2f794956bec0","Type":"ContainerDied","Data":"811c004bfa73708d52bf48a79f537770e811f1d9f1efed3e2e554273c2fc5cf9"} Jul 07 00:21:58.842717 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.842716 2568 scope.go:117] "RemoveContainer" containerID="a16799fe2839138c72d16703287d06c40641c50870bb0c0c1d27832aa75ca20a" Jul 07 00:21:58.846124 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.846104 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-token\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:21:58.846124 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.846124 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-envoy\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:21:58.846274 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.846134 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4blg5\" (UniqueName: \"kubernetes.io/projected/6ea199d3-a76f-4e1e-82dd-2f794956bec0-kube-api-access-4blg5\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:21:58.846274 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.846158 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-workload-socket\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:21:58.846274 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.846169 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-data\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:21:58.846274 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.846177 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea199d3-a76f-4e1e-82dd-2f794956bec0-istio-podinfo\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:21:58.851677 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.851661 2568 scope.go:117] "RemoveContainer" containerID="a16799fe2839138c72d16703287d06c40641c50870bb0c0c1d27832aa75ca20a" Jul 07 00:21:58.851942 ip-10-0-129-125 kubenswrapper[2568]: E0707 00:21:58.851923 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16799fe2839138c72d16703287d06c40641c50870bb0c0c1d27832aa75ca20a\": container with ID starting with a16799fe2839138c72d16703287d06c40641c50870bb0c0c1d27832aa75ca20a not found: ID does not exist" containerID="a16799fe2839138c72d16703287d06c40641c50870bb0c0c1d27832aa75ca20a" Jul 07 00:21:58.851981 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.851949 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16799fe2839138c72d16703287d06c40641c50870bb0c0c1d27832aa75ca20a"} err="failed to get container status \"a16799fe2839138c72d16703287d06c40641c50870bb0c0c1d27832aa75ca20a\": rpc error: code = NotFound desc = could not find container \"a16799fe2839138c72d16703287d06c40641c50870bb0c0c1d27832aa75ca20a\": container with ID starting with a16799fe2839138c72d16703287d06c40641c50870bb0c0c1d27832aa75ca20a not found: ID does not exist" Jul 07 00:21:58.864064 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.864015 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj"] Jul 07 00:21:58.868310 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:58.868289 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7a16f626-openshift-default-5f58778469-8gwdj"] Jul 07 00:21:59.487764 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:21:59.487735 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea199d3-a76f-4e1e-82dd-2f794956bec0" path="/var/lib/kubelet/pods/6ea199d3-a76f-4e1e-82dd-2f794956bec0/volumes" Jul 07 00:22:07.419495 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:22:07.419457 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:22:16.224834 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:22:16.224800 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:23:39.318168 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:23:39.318134 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:23:47.976356 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:23:47.976315 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:23:57.914606 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:23:57.914572 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:24:10.818896 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:10.818861 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:24:33.562024 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.561993 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28"] Jul 07 00:24:33.562508 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.562457 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea199d3-a76f-4e1e-82dd-2f794956bec0" containerName="istio-proxy" Jul 07 00:24:33.562508 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.562474 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea199d3-a76f-4e1e-82dd-2f794956bec0" containerName="istio-proxy" Jul 07 00:24:33.562633 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.562543 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea199d3-a76f-4e1e-82dd-2f794956bec0" containerName="istio-proxy" Jul 07 00:24:33.565363 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.565345 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.567936 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.567919 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0dd24d04-openshift-default-dockercfg-c2svw\"" Jul 07 00:24:33.578184 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.578155 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28"] Jul 07 00:24:33.746520 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.746485 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istiod-ca-cert\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.746520 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.746524 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6r64\" (UniqueName: \"kubernetes.io/projected/500ef3ce-1487-4c8f-a9b6-f6115b72555a-kube-api-access-p6r64\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.746722 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.746545 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-workload-socket\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.746722 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.746611 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-credential-socket\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.746722 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.746672 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-envoy\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.746722 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.746690 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-workload-certs\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.746852 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.746738 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-podinfo\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.746852 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.746763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-data\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.746852 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.746792 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-token\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.847783 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.847703 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-envoy\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.847783 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.847737 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-workload-certs\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.847783 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.847765 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-podinfo\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.847783 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.847781 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-data\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.848124 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.847798 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-token\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.848124 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.847824 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istiod-ca-cert\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.848124 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.847849 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6r64\" (UniqueName: \"kubernetes.io/projected/500ef3ce-1487-4c8f-a9b6-f6115b72555a-kube-api-access-p6r64\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.848124 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.847890 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-workload-socket\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.848124 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.847920 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-credential-socket\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.848318 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.848246 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-workload-certs\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.848318 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.848266 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-data\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.848592 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.848569 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-workload-socket\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.848648 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.848595 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-credential-socket\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.848684 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.848660 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istiod-ca-cert\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.850097 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.850072 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-envoy\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.850283 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.850266 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-podinfo\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.856219 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.856183 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-token\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.856357 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.856339 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6r64\" (UniqueName: \"kubernetes.io/projected/500ef3ce-1487-4c8f-a9b6-f6115b72555a-kube-api-access-p6r64\") pod \"e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:33.877189 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:33.877161 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:34.012520 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:34.012491 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28"] Jul 07 00:24:34.014476 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:24:34.014450 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500ef3ce_1487_4c8f_a9b6_f6115b72555a.slice/crio-b7d65492e2d459c47352a3366d5d9e9c8e7466a497c404b69509c527d7f69571 WatchSource:0}: Error finding container b7d65492e2d459c47352a3366d5d9e9c8e7466a497c404b69509c527d7f69571: Status 404 returned error can't find the container with id b7d65492e2d459c47352a3366d5d9e9c8e7466a497c404b69509c527d7f69571 Jul 07 00:24:34.016617 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:34.016579 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:24:34.016697 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:34.016668 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:24:34.016735 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:34.016708 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:24:34.589145 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:34.589102 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" event={"ID":"500ef3ce-1487-4c8f-a9b6-f6115b72555a","Type":"ContainerStarted","Data":"f18badcf0ca0ec108187c56b98a1e1b8e10f3e51f5b07106af9b5f13f8c85a4e"} Jul 07 00:24:34.589555 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:34.589187 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" event={"ID":"500ef3ce-1487-4c8f-a9b6-f6115b72555a","Type":"ContainerStarted","Data":"b7d65492e2d459c47352a3366d5d9e9c8e7466a497c404b69509c527d7f69571"} Jul 07 00:24:34.608942 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:34.608896 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" podStartSLOduration=1.608881523 podStartE2EDuration="1.608881523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:24:34.606537585 +0000 UTC m=+1697.746994642" watchObservedRunningTime="2026-07-07 00:24:34.608881523 +0000 UTC m=+1697.749338579" Jul 07 00:24:34.877641 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:34.877563 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:35.883292 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:35.883261 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:36.598864 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:36.598833 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:36.599653 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:36.599633 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:41.925001 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:41.924909 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:24:49.666489 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:49.666381 2568 generic.go:356] "Generic (PLEG): container finished" podID="500ef3ce-1487-4c8f-a9b6-f6115b72555a" containerID="f18badcf0ca0ec108187c56b98a1e1b8e10f3e51f5b07106af9b5f13f8c85a4e" exitCode=0 Jul 07 00:24:49.666489 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:49.666464 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" event={"ID":"500ef3ce-1487-4c8f-a9b6-f6115b72555a","Type":"ContainerDied","Data":"f18badcf0ca0ec108187c56b98a1e1b8e10f3e51f5b07106af9b5f13f8c85a4e"} Jul 07 00:24:49.667023 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:49.666891 2568 scope.go:117] "RemoveContainer" containerID="f18badcf0ca0ec108187c56b98a1e1b8e10f3e51f5b07106af9b5f13f8c85a4e" Jul 07 00:24:49.667839 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:49.667813 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:24:49.667940 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:49.667883 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:24:49.667940 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:49.667916 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:24:50.611571 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:50.611536 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:24:50.673189 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:50.673152 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" event={"ID":"500ef3ce-1487-4c8f-a9b6-f6115b72555a","Type":"ContainerStarted","Data":"8337380758f8d035a7a0cae00ffd5869b01e24e6124173e2bdc031b3ad154808"} Jul 07 00:24:50.878159 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:50.878077 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:50.879179 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:50.879152 2568 patch_prober.go:28] interesting pod/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 07 00:24:50.879334 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:50.879217 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" podUID="500ef3ce-1487-4c8f-a9b6-f6115b72555a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 07 00:24:51.599001 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:51.598971 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:51.878461 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:51.878354 2568 patch_prober.go:28] interesting pod/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 07 00:24:51.878461 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:51.878407 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" podUID="500ef3ce-1487-4c8f-a9b6-f6115b72555a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 07 00:24:52.877623 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:52.877593 2568 patch_prober.go:28] interesting pod/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 07 00:24:52.877798 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:52.877642 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" podUID="500ef3ce-1487-4c8f-a9b6-f6115b72555a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 07 00:24:53.878548 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:53.878513 2568 patch_prober.go:28] interesting pod/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 07 00:24:53.878983 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:53.878578 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" podUID="500ef3ce-1487-4c8f-a9b6-f6115b72555a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 07 00:24:54.881733 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:54.881704 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:24:54.882813 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:24:54.882792 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:25:06.023452 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:25:06.023401 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:25:21.128500 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:25:21.128464 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:25:58.808158 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:25:58.808125 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:26:02.821155 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:02.821117 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28"] Jul 07 00:26:02.821563 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:02.821350 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" podUID="500ef3ce-1487-4c8f-a9b6-f6115b72555a" containerName="istio-proxy" containerID="cri-o://8337380758f8d035a7a0cae00ffd5869b01e24e6124173e2bdc031b3ad154808" gracePeriod=30 Jul 07 00:26:08.039742 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.039711 2568 generic.go:356] "Generic (PLEG): container finished" podID="500ef3ce-1487-4c8f-a9b6-f6115b72555a" containerID="8337380758f8d035a7a0cae00ffd5869b01e24e6124173e2bdc031b3ad154808" exitCode=0 Jul 07 00:26:08.040133 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.039750 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" event={"ID":"500ef3ce-1487-4c8f-a9b6-f6115b72555a","Type":"ContainerDied","Data":"8337380758f8d035a7a0cae00ffd5869b01e24e6124173e2bdc031b3ad154808"} Jul 07 00:26:08.040133 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.039796 2568 scope.go:117] "RemoveContainer" containerID="f18badcf0ca0ec108187c56b98a1e1b8e10f3e51f5b07106af9b5f13f8c85a4e" Jul 07 00:26:08.076439 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.076399 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:26:08.170668 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.170586 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-podinfo\") pod \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " Jul 07 00:26:08.170668 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.170640 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-envoy\") pod \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " Jul 07 00:26:08.170872 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.170689 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-workload-socket\") pod \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " Jul 07 00:26:08.170872 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.170726 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-data\") pod \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " Jul 07 00:26:08.170872 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.170761 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-workload-certs\") pod \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " Jul 07 00:26:08.170872 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.170803 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6r64\" (UniqueName: \"kubernetes.io/projected/500ef3ce-1487-4c8f-a9b6-f6115b72555a-kube-api-access-p6r64\") pod \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " Jul 07 00:26:08.170872 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.170827 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-credential-socket\") pod \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " Jul 07 00:26:08.170872 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.170851 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-token\") pod \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " Jul 07 00:26:08.171149 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.170879 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istiod-ca-cert\") pod \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\" (UID: \"500ef3ce-1487-4c8f-a9b6-f6115b72555a\") " Jul 07 00:26:08.171149 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.170939 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "500ef3ce-1487-4c8f-a9b6-f6115b72555a" (UID: "500ef3ce-1487-4c8f-a9b6-f6115b72555a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:26:08.171149 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.171092 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "500ef3ce-1487-4c8f-a9b6-f6115b72555a" (UID: "500ef3ce-1487-4c8f-a9b6-f6115b72555a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:26:08.171149 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.171143 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-workload-socket\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:26:08.171360 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.171177 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "500ef3ce-1487-4c8f-a9b6-f6115b72555a" (UID: "500ef3ce-1487-4c8f-a9b6-f6115b72555a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:26:08.171476 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.171457 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "500ef3ce-1487-4c8f-a9b6-f6115b72555a" (UID: "500ef3ce-1487-4c8f-a9b6-f6115b72555a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:26:08.171723 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.171699 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-data" (OuterVolumeSpecName: "istio-data") pod "500ef3ce-1487-4c8f-a9b6-f6115b72555a" (UID: "500ef3ce-1487-4c8f-a9b6-f6115b72555a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:26:08.172931 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.172902 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500ef3ce-1487-4c8f-a9b6-f6115b72555a-kube-api-access-p6r64" (OuterVolumeSpecName: "kube-api-access-p6r64") pod "500ef3ce-1487-4c8f-a9b6-f6115b72555a" (UID: "500ef3ce-1487-4c8f-a9b6-f6115b72555a"). InnerVolumeSpecName "kube-api-access-p6r64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:26:08.172931 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.172916 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "500ef3ce-1487-4c8f-a9b6-f6115b72555a" (UID: "500ef3ce-1487-4c8f-a9b6-f6115b72555a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:26:08.173202 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.173189 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "500ef3ce-1487-4c8f-a9b6-f6115b72555a" (UID: "500ef3ce-1487-4c8f-a9b6-f6115b72555a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:26:08.173324 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.173303 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-token" (OuterVolumeSpecName: "istio-token") pod "500ef3ce-1487-4c8f-a9b6-f6115b72555a" (UID: "500ef3ce-1487-4c8f-a9b6-f6115b72555a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:26:08.272299 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.272267 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-podinfo\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:26:08.272299 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.272291 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-envoy\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:26:08.272299 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.272300 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-data\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:26:08.272299 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.272307 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-workload-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:26:08.272588 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.272315 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6r64\" (UniqueName: \"kubernetes.io/projected/500ef3ce-1487-4c8f-a9b6-f6115b72555a-kube-api-access-p6r64\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:26:08.272588 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.272324 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500ef3ce-1487-4c8f-a9b6-f6115b72555a-credential-socket\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:26:08.272588 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.272332 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istio-token\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:26:08.272588 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:08.272340 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500ef3ce-1487-4c8f-a9b6-f6115b72555a-istiod-ca-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:26:09.045138 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:09.045061 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" event={"ID":"500ef3ce-1487-4c8f-a9b6-f6115b72555a","Type":"ContainerDied","Data":"b7d65492e2d459c47352a3366d5d9e9c8e7466a497c404b69509c527d7f69571"} Jul 07 00:26:09.045138 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:09.045087 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28" Jul 07 00:26:09.045138 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:09.045120 2568 scope.go:117] "RemoveContainer" containerID="8337380758f8d035a7a0cae00ffd5869b01e24e6124173e2bdc031b3ad154808" Jul 07 00:26:09.069091 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:09.069058 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28"] Jul 07 00:26:09.075078 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:09.075055 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0dd24d04-openshift-default-68dfb8bbc9-rcf28"] Jul 07 00:26:09.485413 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:26:09.485381 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="500ef3ce-1487-4c8f-a9b6-f6115b72555a" path="/var/lib/kubelet/pods/500ef3ce-1487-4c8f-a9b6-f6115b72555a/volumes" Jul 07 00:27:06.155544 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.155512 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5"] Jul 07 00:27:06.155970 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.155798 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="500ef3ce-1487-4c8f-a9b6-f6115b72555a" containerName="istio-proxy" Jul 07 00:27:06.155970 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.155811 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="500ef3ce-1487-4c8f-a9b6-f6115b72555a" containerName="istio-proxy" Jul 07 00:27:06.155970 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.155821 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="500ef3ce-1487-4c8f-a9b6-f6115b72555a" containerName="istio-proxy" Jul 07 00:27:06.155970 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.155826 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="500ef3ce-1487-4c8f-a9b6-f6115b72555a" containerName="istio-proxy" Jul 07 00:27:06.155970 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.155890 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="500ef3ce-1487-4c8f-a9b6-f6115b72555a" containerName="istio-proxy" Jul 07 00:27:06.155970 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.155899 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="500ef3ce-1487-4c8f-a9b6-f6115b72555a" containerName="istio-proxy" Jul 07 00:27:06.158030 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.158015 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.162232 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.162211 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3d04a9-openshift-default-dockercfg-bj6qd\"" Jul 07 00:27:06.175874 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.175839 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5"] Jul 07 00:27:06.225529 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.225505 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-workload-socket\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.225631 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.225568 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-token\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.225693 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.225632 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-credential-socket\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.225745 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.225708 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-workload-certs\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.225790 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.225764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-envoy\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.225844 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.225828 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e34369-83e6-48e3-9c20-a92f45c67b90-istiod-ca-cert\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.225894 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.225861 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-data\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.225933 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.225910 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-podinfo\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.225974 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.225960 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbx88\" (UniqueName: \"kubernetes.io/projected/45e34369-83e6-48e3-9c20-a92f45c67b90-kube-api-access-tbx88\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.326588 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.326561 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-token\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.326731 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.326602 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-credential-socket\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.326731 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.326636 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-workload-certs\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.326731 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.326674 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-envoy\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.326897 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.326738 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e34369-83e6-48e3-9c20-a92f45c67b90-istiod-ca-cert\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.326897 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.326770 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-data\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.326897 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.326797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-podinfo\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.326897 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.326829 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbx88\" (UniqueName: \"kubernetes.io/projected/45e34369-83e6-48e3-9c20-a92f45c67b90-kube-api-access-tbx88\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.326897 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.326870 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-workload-socket\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.327151 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.327021 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-credential-socket\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.327151 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.327104 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-workload-certs\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.327484 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.327459 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-data\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.327607 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.327520 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-workload-socket\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.327726 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.327703 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e34369-83e6-48e3-9c20-a92f45c67b90-istiod-ca-cert\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.329257 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.329235 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-envoy\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.329375 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.329355 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-podinfo\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.335167 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.335138 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbx88\" (UniqueName: \"kubernetes.io/projected/45e34369-83e6-48e3-9c20-a92f45c67b90-kube-api-access-tbx88\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.335167 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.335162 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-token\") pod \"e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.468604 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.468580 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:06.606543 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.606517 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5"] Jul 07 00:27:06.608064 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:27:06.608037 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e34369_83e6_48e3_9c20_a92f45c67b90.slice/crio-7007c2effd5e7211bb51b5bb821ed2d617a83aa21910a35ea9e9703c267589bf WatchSource:0}: Error finding container 7007c2effd5e7211bb51b5bb821ed2d617a83aa21910a35ea9e9703c267589bf: Status 404 returned error can't find the container with id 7007c2effd5e7211bb51b5bb821ed2d617a83aa21910a35ea9e9703c267589bf Jul 07 00:27:06.610035 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.610003 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:27:06.610130 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.610067 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:27:06.610130 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:06.610100 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:27:07.325648 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:07.325615 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" event={"ID":"45e34369-83e6-48e3-9c20-a92f45c67b90","Type":"ContainerStarted","Data":"a79f33519e73d399c19ae811456e6330ef752a27ad7b69102e0be463dabc2cb4"} Jul 07 00:27:07.326124 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:07.325678 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" event={"ID":"45e34369-83e6-48e3-9c20-a92f45c67b90","Type":"ContainerStarted","Data":"7007c2effd5e7211bb51b5bb821ed2d617a83aa21910a35ea9e9703c267589bf"} Jul 07 00:27:07.346040 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:07.345999 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" podStartSLOduration=1.345985181 podStartE2EDuration="1.345985181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:27:07.344463856 +0000 UTC m=+1850.484920916" watchObservedRunningTime="2026-07-07 00:27:07.345985181 +0000 UTC m=+1850.486442229" Jul 07 00:27:07.468862 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:07.468834 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:07.473812 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:07.473787 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:08.330414 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:08.330387 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:08.331370 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:08.331350 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:09.821583 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:09.821550 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:27:21.521707 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:21.521679 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:27:24.201862 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:24.201828 2568 patch_prober.go:28] interesting pod/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:33528->10.133.0.41:15021: read: connection reset by peer" start-of-body= Jul 07 00:27:24.202256 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:24.201893 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" podUID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:33528->10.133.0.41:15021: read: connection reset by peer" Jul 07 00:27:24.407525 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:24.407446 2568 generic.go:356] "Generic (PLEG): container finished" podID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerID="a79f33519e73d399c19ae811456e6330ef752a27ad7b69102e0be463dabc2cb4" exitCode=0 Jul 07 00:27:24.407672 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:24.407518 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" event={"ID":"45e34369-83e6-48e3-9c20-a92f45c67b90","Type":"ContainerDied","Data":"a79f33519e73d399c19ae811456e6330ef752a27ad7b69102e0be463dabc2cb4"} Jul 07 00:27:24.407920 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:24.407905 2568 scope.go:117] "RemoveContainer" containerID="a79f33519e73d399c19ae811456e6330ef752a27ad7b69102e0be463dabc2cb4" Jul 07 00:27:24.408902 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:24.408878 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:27:24.408958 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:24.408944 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:27:24.409000 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:24.408975 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:27:25.413805 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:25.413770 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" event={"ID":"45e34369-83e6-48e3-9c20-a92f45c67b90","Type":"ContainerStarted","Data":"10325c15f997fc342d5736bed817fa9f086f589acf443e55ebfb377af2721816"} Jul 07 00:27:25.469506 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:25.469467 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:25.469729 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:25.469706 2568 patch_prober.go:28] interesting pod/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 07 00:27:25.469773 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:25.469755 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" podUID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 07 00:27:26.469469 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:26.469411 2568 patch_prober.go:28] interesting pod/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 07 00:27:26.469873 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:26.469488 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" podUID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 07 00:27:27.469367 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:27.469326 2568 patch_prober.go:28] interesting pod/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 07 00:27:27.469571 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:27.469384 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" podUID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 07 00:27:28.469789 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:28.469753 2568 patch_prober.go:28] interesting pod/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 07 00:27:28.470221 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:28.469806 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" podUID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 07 00:27:29.472383 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:29.472349 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:29.472772 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:29.472618 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:29.473316 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:29.473296 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:27:43.815893 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:43.815811 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:27:54.725819 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:27:54.725783 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:28:01.512186 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.512151 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2"] Jul 07 00:28:01.514390 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.514370 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.520110 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.520085 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3d04a9\"/\"default-dockercfg-lx7gp\"" Jul 07 00:28:01.520246 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.520084 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3d04a9\"/\"test-model-3d04a9-kserve-self-signed-certs\"" Jul 07 00:28:01.520246 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.520091 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3d04a9\"/\"kube-root-ca.crt\"" Jul 07 00:28:01.520410 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.520394 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3d04a9\"/\"openshift-service-ca.crt\"" Jul 07 00:28:01.534371 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.534348 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2"] Jul 07 00:28:01.681985 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.681951 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-model-cache\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.681985 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.681984 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-tmp-dir\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.682202 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.682056 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvkdx\" (UniqueName: \"kubernetes.io/projected/11f53d20-5532-4b2c-899a-e97656af0ac3-kube-api-access-kvkdx\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.682202 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.682083 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-home\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.682202 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.682145 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f53d20-5532-4b2c-899a-e97656af0ac3-tls-certs\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.682202 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.682178 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-dshm\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.783391 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.783306 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-model-cache\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.783391 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.783341 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-tmp-dir\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.783391 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.783389 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvkdx\" (UniqueName: \"kubernetes.io/projected/11f53d20-5532-4b2c-899a-e97656af0ac3-kube-api-access-kvkdx\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.783697 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.783411 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-home\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.783697 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.783448 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f53d20-5532-4b2c-899a-e97656af0ac3-tls-certs\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.783697 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.783474 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-dshm\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.783867 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.783780 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-model-cache\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.783867 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.783790 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-home\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.783867 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.783836 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-tmp-dir\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.785709 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.785685 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-dshm\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.786114 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.786093 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f53d20-5532-4b2c-899a-e97656af0ac3-tls-certs\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.792233 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.792214 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvkdx\" (UniqueName: \"kubernetes.io/projected/11f53d20-5532-4b2c-899a-e97656af0ac3-kube-api-access-kvkdx\") pod \"test-model-3d04a9-kserve-5958f76947-bpnx2\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.824246 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.824217 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:01.960117 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:01.960093 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2"] Jul 07 00:28:01.962034 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:28:01.962003 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11f53d20_5532_4b2c_899a_e97656af0ac3.slice/crio-659d8dc8aa5606686d2c92924022dd2597ba58f98447cbcedcc26dd40f86398a WatchSource:0}: Error finding container 659d8dc8aa5606686d2c92924022dd2597ba58f98447cbcedcc26dd40f86398a: Status 404 returned error can't find the container with id 659d8dc8aa5606686d2c92924022dd2597ba58f98447cbcedcc26dd40f86398a Jul 07 00:28:02.605128 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:02.605092 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" event={"ID":"11f53d20-5532-4b2c-899a-e97656af0ac3","Type":"ContainerStarted","Data":"be9b07871c1597fb989e6026d55f56a7c12504f17448f1d400f1aeb53c99b95b"} Jul 07 00:28:02.605128 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:02.605132 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" event={"ID":"11f53d20-5532-4b2c-899a-e97656af0ac3","Type":"ContainerStarted","Data":"659d8dc8aa5606686d2c92924022dd2597ba58f98447cbcedcc26dd40f86398a"} Jul 07 00:28:06.708537 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:06.708486 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" podStartSLOduration=5.708469336 podStartE2EDuration="5.708469336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:28:02.62157627 +0000 UTC m=+1905.762033327" watchObservedRunningTime="2026-07-07 00:28:06.708469336 +0000 UTC m=+1909.848926393" Jul 07 00:28:06.709674 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:06.709656 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:28:11.824451 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:11.824409 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:11.824883 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:11.824458 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:11.836930 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:11.836908 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:12.661586 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:12.661562 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:20.313281 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:20.313247 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:28:23.581702 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:23.581671 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2"] Jul 07 00:28:37.433802 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:37.433769 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:28:38.600943 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.600887 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" podUID="11f53d20-5532-4b2c-899a-e97656af0ac3" containerName="main" containerID="cri-o://be9b07871c1597fb989e6026d55f56a7c12504f17448f1d400f1aeb53c99b95b" gracePeriod=45 Jul 07 00:28:38.776223 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.776191 2568 generic.go:356] "Generic (PLEG): container finished" podID="11f53d20-5532-4b2c-899a-e97656af0ac3" containerID="be9b07871c1597fb989e6026d55f56a7c12504f17448f1d400f1aeb53c99b95b" exitCode=0 Jul 07 00:28:38.776381 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.776277 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" event={"ID":"11f53d20-5532-4b2c-899a-e97656af0ac3","Type":"ContainerDied","Data":"be9b07871c1597fb989e6026d55f56a7c12504f17448f1d400f1aeb53c99b95b"} Jul 07 00:28:38.846890 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.846866 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:38.963505 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.963478 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-model-cache\") pod \"11f53d20-5532-4b2c-899a-e97656af0ac3\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " Jul 07 00:28:38.963616 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.963536 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-tmp-dir\") pod \"11f53d20-5532-4b2c-899a-e97656af0ac3\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " Jul 07 00:28:38.963616 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.963562 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f53d20-5532-4b2c-899a-e97656af0ac3-tls-certs\") pod \"11f53d20-5532-4b2c-899a-e97656af0ac3\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " Jul 07 00:28:38.963616 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.963610 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-dshm\") pod \"11f53d20-5532-4b2c-899a-e97656af0ac3\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " Jul 07 00:28:38.963721 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.963645 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvkdx\" (UniqueName: \"kubernetes.io/projected/11f53d20-5532-4b2c-899a-e97656af0ac3-kube-api-access-kvkdx\") pod \"11f53d20-5532-4b2c-899a-e97656af0ac3\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " Jul 07 00:28:38.963772 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.963717 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-model-cache" (OuterVolumeSpecName: "model-cache") pod "11f53d20-5532-4b2c-899a-e97656af0ac3" (UID: "11f53d20-5532-4b2c-899a-e97656af0ac3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:38.963772 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.963725 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-home\") pod \"11f53d20-5532-4b2c-899a-e97656af0ac3\" (UID: \"11f53d20-5532-4b2c-899a-e97656af0ac3\") " Jul 07 00:28:38.963876 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.963847 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "11f53d20-5532-4b2c-899a-e97656af0ac3" (UID: "11f53d20-5532-4b2c-899a-e97656af0ac3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:38.963966 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.963948 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-home" (OuterVolumeSpecName: "home") pod "11f53d20-5532-4b2c-899a-e97656af0ac3" (UID: "11f53d20-5532-4b2c-899a-e97656af0ac3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:38.964034 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.964016 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:28:38.964034 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.964028 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:28:38.964133 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.964039 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:28:38.965736 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.965711 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-dshm" (OuterVolumeSpecName: "dshm") pod "11f53d20-5532-4b2c-899a-e97656af0ac3" (UID: "11f53d20-5532-4b2c-899a-e97656af0ac3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:38.965839 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.965762 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f53d20-5532-4b2c-899a-e97656af0ac3-kube-api-access-kvkdx" (OuterVolumeSpecName: "kube-api-access-kvkdx") pod "11f53d20-5532-4b2c-899a-e97656af0ac3" (UID: "11f53d20-5532-4b2c-899a-e97656af0ac3"). InnerVolumeSpecName "kube-api-access-kvkdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:28:38.965839 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:38.965829 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f53d20-5532-4b2c-899a-e97656af0ac3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11f53d20-5532-4b2c-899a-e97656af0ac3" (UID: "11f53d20-5532-4b2c-899a-e97656af0ac3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:28:39.065325 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:39.065303 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f53d20-5532-4b2c-899a-e97656af0ac3-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:28:39.065325 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:39.065323 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f53d20-5532-4b2c-899a-e97656af0ac3-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:28:39.065474 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:39.065332 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvkdx\" (UniqueName: \"kubernetes.io/projected/11f53d20-5532-4b2c-899a-e97656af0ac3-kube-api-access-kvkdx\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:28:39.782276 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:39.782210 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" event={"ID":"11f53d20-5532-4b2c-899a-e97656af0ac3","Type":"ContainerDied","Data":"659d8dc8aa5606686d2c92924022dd2597ba58f98447cbcedcc26dd40f86398a"} Jul 07 00:28:39.782276 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:39.782244 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2" Jul 07 00:28:39.782276 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:39.782258 2568 scope.go:117] "RemoveContainer" containerID="be9b07871c1597fb989e6026d55f56a7c12504f17448f1d400f1aeb53c99b95b" Jul 07 00:28:39.797643 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:39.797623 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2"] Jul 07 00:28:39.800361 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:39.800339 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3d04a9/test-model-3d04a9-kserve-5958f76947-bpnx2"] Jul 07 00:28:41.486735 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:28:41.486705 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11f53d20-5532-4b2c-899a-e97656af0ac3" path="/var/lib/kubelet/pods/11f53d20-5532-4b2c-899a-e97656af0ac3/volumes" Jul 07 00:29:10.734625 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:10.734556 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:29:13.455755 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:13.455712 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5"] Jul 07 00:29:13.456248 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:13.455986 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" podUID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerName="istio-proxy" containerID="cri-o://10325c15f997fc342d5736bed817fa9f086f589acf443e55ebfb377af2721816" gracePeriod=30 Jul 07 00:29:14.475165 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:14.475128 2568 patch_prober.go:28] interesting pod/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 00:29:14.475565 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:14.475205 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" podUID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 00:29:18.699448 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.699405 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:29:18.864710 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.864623 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e34369-83e6-48e3-9c20-a92f45c67b90-istiod-ca-cert\") pod \"45e34369-83e6-48e3-9c20-a92f45c67b90\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " Jul 07 00:29:18.864710 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.864660 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-envoy\") pod \"45e34369-83e6-48e3-9c20-a92f45c67b90\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " Jul 07 00:29:18.864710 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.864693 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-data\") pod \"45e34369-83e6-48e3-9c20-a92f45c67b90\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " Jul 07 00:29:18.864710 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.864729 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbx88\" (UniqueName: \"kubernetes.io/projected/45e34369-83e6-48e3-9c20-a92f45c67b90-kube-api-access-tbx88\") pod \"45e34369-83e6-48e3-9c20-a92f45c67b90\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " Jul 07 00:29:18.864710 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.864758 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-credential-socket\") pod \"45e34369-83e6-48e3-9c20-a92f45c67b90\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " Jul 07 00:29:18.865150 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.864819 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-workload-certs\") pod \"45e34369-83e6-48e3-9c20-a92f45c67b90\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " Jul 07 00:29:18.865150 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.864868 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-token\") pod \"45e34369-83e6-48e3-9c20-a92f45c67b90\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " Jul 07 00:29:18.865150 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.864893 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-workload-socket\") pod \"45e34369-83e6-48e3-9c20-a92f45c67b90\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " Jul 07 00:29:18.865150 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.864914 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-podinfo\") pod \"45e34369-83e6-48e3-9c20-a92f45c67b90\" (UID: \"45e34369-83e6-48e3-9c20-a92f45c67b90\") " Jul 07 00:29:18.865150 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.865042 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e34369-83e6-48e3-9c20-a92f45c67b90-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45e34369-83e6-48e3-9c20-a92f45c67b90" (UID: "45e34369-83e6-48e3-9c20-a92f45c67b90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:29:18.865464 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.865187 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e34369-83e6-48e3-9c20-a92f45c67b90-istiod-ca-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:18.865464 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.865300 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45e34369-83e6-48e3-9c20-a92f45c67b90" (UID: "45e34369-83e6-48e3-9c20-a92f45c67b90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:29:18.865613 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.865495 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45e34369-83e6-48e3-9c20-a92f45c67b90" (UID: "45e34369-83e6-48e3-9c20-a92f45c67b90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:29:18.865613 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.865516 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45e34369-83e6-48e3-9c20-a92f45c67b90" (UID: "45e34369-83e6-48e3-9c20-a92f45c67b90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:29:18.865728 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.865678 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-data" (OuterVolumeSpecName: "istio-data") pod "45e34369-83e6-48e3-9c20-a92f45c67b90" (UID: "45e34369-83e6-48e3-9c20-a92f45c67b90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:29:18.867048 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.867025 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45e34369-83e6-48e3-9c20-a92f45c67b90" (UID: "45e34369-83e6-48e3-9c20-a92f45c67b90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:29:18.867331 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.867302 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-token" (OuterVolumeSpecName: "istio-token") pod "45e34369-83e6-48e3-9c20-a92f45c67b90" (UID: "45e34369-83e6-48e3-9c20-a92f45c67b90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:29:18.867401 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.867343 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e34369-83e6-48e3-9c20-a92f45c67b90-kube-api-access-tbx88" (OuterVolumeSpecName: "kube-api-access-tbx88") pod "45e34369-83e6-48e3-9c20-a92f45c67b90" (UID: "45e34369-83e6-48e3-9c20-a92f45c67b90"). InnerVolumeSpecName "kube-api-access-tbx88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:29:18.867401 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.867387 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45e34369-83e6-48e3-9c20-a92f45c67b90" (UID: "45e34369-83e6-48e3-9c20-a92f45c67b90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:29:18.965828 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.965794 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-envoy\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:18.965828 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.965822 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-data\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:18.965828 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.965833 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbx88\" (UniqueName: \"kubernetes.io/projected/45e34369-83e6-48e3-9c20-a92f45c67b90-kube-api-access-tbx88\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:18.966040 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.965845 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-credential-socket\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:18.966040 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.965857 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-workload-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:18.966040 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.965867 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-token\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:18.966040 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.965879 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e34369-83e6-48e3-9c20-a92f45c67b90-workload-socket\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:18.966040 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.965889 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e34369-83e6-48e3-9c20-a92f45c67b90-istio-podinfo\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:18.979006 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.978974 2568 generic.go:356] "Generic (PLEG): container finished" podID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerID="10325c15f997fc342d5736bed817fa9f086f589acf443e55ebfb377af2721816" exitCode=0 Jul 07 00:29:18.979146 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.979039 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" Jul 07 00:29:18.979146 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.979066 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" event={"ID":"45e34369-83e6-48e3-9c20-a92f45c67b90","Type":"ContainerDied","Data":"10325c15f997fc342d5736bed817fa9f086f589acf443e55ebfb377af2721816"} Jul 07 00:29:18.979233 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.979153 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5" event={"ID":"45e34369-83e6-48e3-9c20-a92f45c67b90","Type":"ContainerDied","Data":"7007c2effd5e7211bb51b5bb821ed2d617a83aa21910a35ea9e9703c267589bf"} Jul 07 00:29:18.979233 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.979182 2568 scope.go:117] "RemoveContainer" containerID="10325c15f997fc342d5736bed817fa9f086f589acf443e55ebfb377af2721816" Jul 07 00:29:18.988228 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.988213 2568 scope.go:117] "RemoveContainer" containerID="a79f33519e73d399c19ae811456e6330ef752a27ad7b69102e0be463dabc2cb4" Jul 07 00:29:18.995977 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.995959 2568 scope.go:117] "RemoveContainer" containerID="10325c15f997fc342d5736bed817fa9f086f589acf443e55ebfb377af2721816" Jul 07 00:29:18.996453 ip-10-0-129-125 kubenswrapper[2568]: E0707 00:29:18.996316 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10325c15f997fc342d5736bed817fa9f086f589acf443e55ebfb377af2721816\": container with ID starting with 10325c15f997fc342d5736bed817fa9f086f589acf443e55ebfb377af2721816 not found: ID does not exist" containerID="10325c15f997fc342d5736bed817fa9f086f589acf443e55ebfb377af2721816" Jul 07 00:29:18.996453 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.996353 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10325c15f997fc342d5736bed817fa9f086f589acf443e55ebfb377af2721816"} err="failed to get container status \"10325c15f997fc342d5736bed817fa9f086f589acf443e55ebfb377af2721816\": rpc error: code = NotFound desc = could not find container \"10325c15f997fc342d5736bed817fa9f086f589acf443e55ebfb377af2721816\": container with ID starting with 10325c15f997fc342d5736bed817fa9f086f589acf443e55ebfb377af2721816 not found: ID does not exist" Jul 07 00:29:18.996453 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.996378 2568 scope.go:117] "RemoveContainer" containerID="a79f33519e73d399c19ae811456e6330ef752a27ad7b69102e0be463dabc2cb4" Jul 07 00:29:18.996899 ip-10-0-129-125 kubenswrapper[2568]: E0707 00:29:18.996869 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79f33519e73d399c19ae811456e6330ef752a27ad7b69102e0be463dabc2cb4\": container with ID starting with a79f33519e73d399c19ae811456e6330ef752a27ad7b69102e0be463dabc2cb4 not found: ID does not exist" containerID="a79f33519e73d399c19ae811456e6330ef752a27ad7b69102e0be463dabc2cb4" Jul 07 00:29:18.997010 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.996904 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79f33519e73d399c19ae811456e6330ef752a27ad7b69102e0be463dabc2cb4"} err="failed to get container status \"a79f33519e73d399c19ae811456e6330ef752a27ad7b69102e0be463dabc2cb4\": rpc error: code = NotFound desc = could not find container \"a79f33519e73d399c19ae811456e6330ef752a27ad7b69102e0be463dabc2cb4\": container with ID starting with a79f33519e73d399c19ae811456e6330ef752a27ad7b69102e0be463dabc2cb4 not found: ID does not exist" Jul 07 00:29:18.999024 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:18.999005 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5"] Jul 07 00:29:19.004033 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:19.004011 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3d04a9-openshift-default-58cd794b45-qwtb5"] Jul 07 00:29:19.486608 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:19.486570 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45e34369-83e6-48e3-9c20-a92f45c67b90" path="/var/lib/kubelet/pods/45e34369-83e6-48e3-9c20-a92f45c67b90/volumes" Jul 07 00:29:23.034084 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.034050 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh"] Jul 07 00:29:23.034555 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.034348 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerName="istio-proxy" Jul 07 00:29:23.034555 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.034359 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerName="istio-proxy" Jul 07 00:29:23.034555 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.034378 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f53d20-5532-4b2c-899a-e97656af0ac3" containerName="main" Jul 07 00:29:23.034555 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.034383 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f53d20-5532-4b2c-899a-e97656af0ac3" containerName="main" Jul 07 00:29:23.034555 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.034467 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f53d20-5532-4b2c-899a-e97656af0ac3" containerName="main" Jul 07 00:29:23.034555 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.034475 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerName="istio-proxy" Jul 07 00:29:23.034555 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.034481 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerName="istio-proxy" Jul 07 00:29:23.037493 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.037476 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.040196 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.040176 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7e3541-openshift-default-dockercfg-57x97\"" Jul 07 00:29:23.049882 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.049860 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh"] Jul 07 00:29:23.094828 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.094797 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-data\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.094992 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.094836 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-workload-socket\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.094992 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.094870 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brpp8\" (UniqueName: \"kubernetes.io/projected/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-kube-api-access-brpp8\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.094992 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.094903 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-token\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.094992 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.094932 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-workload-certs\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.095148 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.095023 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istiod-ca-cert\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.095148 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.095060 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-envoy\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.095148 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.095093 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-credential-socket\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.095148 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.095132 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-podinfo\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.196509 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.196476 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brpp8\" (UniqueName: \"kubernetes.io/projected/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-kube-api-access-brpp8\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.196716 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.196516 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-token\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.196716 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.196546 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-workload-certs\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.196716 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.196597 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istiod-ca-cert\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.196716 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.196629 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-envoy\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.196716 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.196661 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-credential-socket\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.196716 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.196689 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-podinfo\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.197027 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.196722 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-data\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.197027 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.196753 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-workload-socket\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.197149 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.197042 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-workload-certs\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.197149 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.197134 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-workload-socket\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.197261 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.197238 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-data\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.197312 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.197289 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-credential-socket\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.197381 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.197362 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istiod-ca-cert\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.198959 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.198932 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-envoy\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.199085 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.199068 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-podinfo\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.204926 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.204901 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brpp8\" (UniqueName: \"kubernetes.io/projected/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-kube-api-access-brpp8\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.205018 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.204955 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-token\") pod \"e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.350306 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.350214 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:23.487969 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.487944 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh"] Jul 07 00:29:23.490033 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:29:23.489999 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5a673e_e8b7_4700_a8bb_1a014c8503ea.slice/crio-04c6bf6ea19f0471861b334b0e7b927e540403bef537077e5b0da0f791aa825e WatchSource:0}: Error finding container 04c6bf6ea19f0471861b334b0e7b927e540403bef537077e5b0da0f791aa825e: Status 404 returned error can't find the container with id 04c6bf6ea19f0471861b334b0e7b927e540403bef537077e5b0da0f791aa825e Jul 07 00:29:23.492145 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.492115 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:29:23.492214 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.492181 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:29:23.492214 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:23.492209 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:29:24.004488 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:24.004455 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" event={"ID":"fd5a673e-e8b7-4700-a8bb-1a014c8503ea","Type":"ContainerStarted","Data":"69fb94cca4aca2caffa696e52455e8e4275ed0b812aad029b80966307a811eca"} Jul 07 00:29:24.004693 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:24.004530 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" event={"ID":"fd5a673e-e8b7-4700-a8bb-1a014c8503ea","Type":"ContainerStarted","Data":"04c6bf6ea19f0471861b334b0e7b927e540403bef537077e5b0da0f791aa825e"} Jul 07 00:29:24.025362 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:24.025316 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" podStartSLOduration=1.025302138 podStartE2EDuration="1.025302138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:29:24.022806274 +0000 UTC m=+1987.163263351" watchObservedRunningTime="2026-07-07 00:29:24.025302138 +0000 UTC m=+1987.165759192" Jul 07 00:29:24.351033 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:24.350944 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:24.355796 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:24.355772 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:25.009491 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:25.009452 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:25.010354 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:25.010336 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:25.507813 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:25.507780 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:29:40.741570 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:40.741533 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:29:41.051666 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:41.051576 2568 patch_prober.go:28] interesting pod/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 00:29:41.051666 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:41.051653 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" podUID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 00:29:42.105501 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:42.105466 2568 generic.go:356] "Generic (PLEG): container finished" podID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" containerID="69fb94cca4aca2caffa696e52455e8e4275ed0b812aad029b80966307a811eca" exitCode=0 Jul 07 00:29:42.105938 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:42.105537 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" event={"ID":"fd5a673e-e8b7-4700-a8bb-1a014c8503ea","Type":"ContainerDied","Data":"69fb94cca4aca2caffa696e52455e8e4275ed0b812aad029b80966307a811eca"} Jul 07 00:29:42.105938 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:42.105911 2568 scope.go:117] "RemoveContainer" containerID="69fb94cca4aca2caffa696e52455e8e4275ed0b812aad029b80966307a811eca" Jul 07 00:29:42.106904 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:42.106876 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:29:42.107031 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:42.106954 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:29:42.107031 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:42.106990 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:29:43.113735 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:43.113698 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" event={"ID":"fd5a673e-e8b7-4700-a8bb-1a014c8503ea","Type":"ContainerStarted","Data":"f27f6970e1f087e9e9d13f39d40fc8e7d52f8edb4a8efb5c6fa84b873a21dd54"} Jul 07 00:29:43.351155 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:43.351107 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:43.351658 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:43.351625 2568 patch_prober.go:28] interesting pod/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 07 00:29:43.351904 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:43.351875 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" podUID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 07 00:29:44.350731 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:44.350690 2568 patch_prober.go:28] interesting pod/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 07 00:29:44.351345 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:44.351315 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" podUID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 07 00:29:45.350966 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:45.350924 2568 patch_prober.go:28] interesting pod/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 07 00:29:45.351617 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:45.351584 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" podUID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 07 00:29:46.350757 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:46.350708 2568 patch_prober.go:28] interesting pod/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 07 00:29:46.350985 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:46.350778 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" podUID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 07 00:29:47.353817 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:47.353788 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:47.354228 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:47.354031 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:47.354606 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:47.354588 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:51.233653 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:51.233619 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:29:53.577905 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:53.577870 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh"] Jul 07 00:29:53.578360 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:53.578047 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" podUID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" containerName="istio-proxy" containerID="cri-o://f27f6970e1f087e9e9d13f39d40fc8e7d52f8edb4a8efb5c6fa84b873a21dd54" gracePeriod=30 Jul 07 00:29:58.824223 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.824201 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:58.896842 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.896762 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brpp8\" (UniqueName: \"kubernetes.io/projected/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-kube-api-access-brpp8\") pod \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " Jul 07 00:29:58.896842 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.896797 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-workload-socket\") pod \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " Jul 07 00:29:58.896842 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.896818 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-credential-socket\") pod \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " Jul 07 00:29:58.897129 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.896846 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-token\") pod \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " Jul 07 00:29:58.897129 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.896874 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-envoy\") pod \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " Jul 07 00:29:58.897129 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.896901 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-workload-certs\") pod \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " Jul 07 00:29:58.897129 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.896931 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-data\") pod \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " Jul 07 00:29:58.897129 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.896949 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-podinfo\") pod \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " Jul 07 00:29:58.897129 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.896990 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istiod-ca-cert\") pod \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\" (UID: \"fd5a673e-e8b7-4700-a8bb-1a014c8503ea\") " Jul 07 00:29:58.897129 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.897092 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd5a673e-e8b7-4700-a8bb-1a014c8503ea" (UID: "fd5a673e-e8b7-4700-a8bb-1a014c8503ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:29:58.897129 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.897126 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd5a673e-e8b7-4700-a8bb-1a014c8503ea" (UID: "fd5a673e-e8b7-4700-a8bb-1a014c8503ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:29:58.897569 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.897257 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-workload-socket\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:58.897569 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.897277 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-credential-socket\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:58.897569 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.897410 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd5a673e-e8b7-4700-a8bb-1a014c8503ea" (UID: "fd5a673e-e8b7-4700-a8bb-1a014c8503ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:29:58.897569 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.897544 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd5a673e-e8b7-4700-a8bb-1a014c8503ea" (UID: "fd5a673e-e8b7-4700-a8bb-1a014c8503ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:29:58.897944 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.897920 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd5a673e-e8b7-4700-a8bb-1a014c8503ea" (UID: "fd5a673e-e8b7-4700-a8bb-1a014c8503ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:29:58.899167 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.899128 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd5a673e-e8b7-4700-a8bb-1a014c8503ea" (UID: "fd5a673e-e8b7-4700-a8bb-1a014c8503ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:29:58.899290 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.899268 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-kube-api-access-brpp8" (OuterVolumeSpecName: "kube-api-access-brpp8") pod "fd5a673e-e8b7-4700-a8bb-1a014c8503ea" (UID: "fd5a673e-e8b7-4700-a8bb-1a014c8503ea"). InnerVolumeSpecName "kube-api-access-brpp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:29:58.899586 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.899555 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd5a673e-e8b7-4700-a8bb-1a014c8503ea" (UID: "fd5a673e-e8b7-4700-a8bb-1a014c8503ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:29:58.899692 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.899583 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd5a673e-e8b7-4700-a8bb-1a014c8503ea" (UID: "fd5a673e-e8b7-4700-a8bb-1a014c8503ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:29:58.998041 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.998007 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-data\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:58.998041 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.998033 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-podinfo\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:58.998041 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.998042 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istiod-ca-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:58.998041 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.998051 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brpp8\" (UniqueName: \"kubernetes.io/projected/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-kube-api-access-brpp8\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:58.998305 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.998059 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-token\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:58.998305 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.998067 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-istio-envoy\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:58.998305 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:58.998074 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5a673e-e8b7-4700-a8bb-1a014c8503ea-workload-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:29:59.195009 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:59.194975 2568 generic.go:356] "Generic (PLEG): container finished" podID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" containerID="f27f6970e1f087e9e9d13f39d40fc8e7d52f8edb4a8efb5c6fa84b873a21dd54" exitCode=0 Jul 07 00:29:59.195204 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:59.195033 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" event={"ID":"fd5a673e-e8b7-4700-a8bb-1a014c8503ea","Type":"ContainerDied","Data":"f27f6970e1f087e9e9d13f39d40fc8e7d52f8edb4a8efb5c6fa84b873a21dd54"} Jul 07 00:29:59.195204 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:59.195039 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" Jul 07 00:29:59.195204 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:59.195081 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh" event={"ID":"fd5a673e-e8b7-4700-a8bb-1a014c8503ea","Type":"ContainerDied","Data":"04c6bf6ea19f0471861b334b0e7b927e540403bef537077e5b0da0f791aa825e"} Jul 07 00:29:59.195204 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:59.195103 2568 scope.go:117] "RemoveContainer" containerID="f27f6970e1f087e9e9d13f39d40fc8e7d52f8edb4a8efb5c6fa84b873a21dd54" Jul 07 00:29:59.204179 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:59.204156 2568 scope.go:117] "RemoveContainer" containerID="69fb94cca4aca2caffa696e52455e8e4275ed0b812aad029b80966307a811eca" Jul 07 00:29:59.212278 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:59.212262 2568 scope.go:117] "RemoveContainer" containerID="f27f6970e1f087e9e9d13f39d40fc8e7d52f8edb4a8efb5c6fa84b873a21dd54" Jul 07 00:29:59.212527 ip-10-0-129-125 kubenswrapper[2568]: E0707 00:29:59.212505 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27f6970e1f087e9e9d13f39d40fc8e7d52f8edb4a8efb5c6fa84b873a21dd54\": container with ID starting with f27f6970e1f087e9e9d13f39d40fc8e7d52f8edb4a8efb5c6fa84b873a21dd54 not found: ID does not exist" containerID="f27f6970e1f087e9e9d13f39d40fc8e7d52f8edb4a8efb5c6fa84b873a21dd54" Jul 07 00:29:59.212582 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:59.212534 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27f6970e1f087e9e9d13f39d40fc8e7d52f8edb4a8efb5c6fa84b873a21dd54"} err="failed to get container status \"f27f6970e1f087e9e9d13f39d40fc8e7d52f8edb4a8efb5c6fa84b873a21dd54\": rpc error: code = NotFound desc = could not find container \"f27f6970e1f087e9e9d13f39d40fc8e7d52f8edb4a8efb5c6fa84b873a21dd54\": container with ID starting with f27f6970e1f087e9e9d13f39d40fc8e7d52f8edb4a8efb5c6fa84b873a21dd54 not found: ID does not exist" Jul 07 00:29:59.212582 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:59.212549 2568 scope.go:117] "RemoveContainer" containerID="69fb94cca4aca2caffa696e52455e8e4275ed0b812aad029b80966307a811eca" Jul 07 00:29:59.212760 ip-10-0-129-125 kubenswrapper[2568]: E0707 00:29:59.212742 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69fb94cca4aca2caffa696e52455e8e4275ed0b812aad029b80966307a811eca\": container with ID starting with 69fb94cca4aca2caffa696e52455e8e4275ed0b812aad029b80966307a811eca not found: ID does not exist" containerID="69fb94cca4aca2caffa696e52455e8e4275ed0b812aad029b80966307a811eca" Jul 07 00:29:59.212805 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:59.212767 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69fb94cca4aca2caffa696e52455e8e4275ed0b812aad029b80966307a811eca"} err="failed to get container status \"69fb94cca4aca2caffa696e52455e8e4275ed0b812aad029b80966307a811eca\": rpc error: code = NotFound desc = could not find container \"69fb94cca4aca2caffa696e52455e8e4275ed0b812aad029b80966307a811eca\": container with ID starting with 69fb94cca4aca2caffa696e52455e8e4275ed0b812aad029b80966307a811eca not found: ID does not exist" Jul 07 00:29:59.216237 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:59.216216 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh"] Jul 07 00:29:59.222528 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:59.222509 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7e3541-openshift-default-595658fc7f-gwxgh"] Jul 07 00:29:59.486153 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:29:59.486086 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" path="/var/lib/kubelet/pods/fd5a673e-e8b7-4700-a8bb-1a014c8503ea/volumes" Jul 07 00:30:00.155413 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.155379 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97"] Jul 07 00:30:00.155784 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.155698 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" containerName="istio-proxy" Jul 07 00:30:00.155784 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.155711 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" containerName="istio-proxy" Jul 07 00:30:00.155784 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.155721 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerName="istio-proxy" Jul 07 00:30:00.155784 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.155727 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e34369-83e6-48e3-9c20-a92f45c67b90" containerName="istio-proxy" Jul 07 00:30:00.155946 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.155792 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" containerName="istio-proxy" Jul 07 00:30:00.155946 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.155801 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" containerName="istio-proxy" Jul 07 00:30:00.160063 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.160045 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97" Jul 07 00:30:00.163246 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.163220 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97"] Jul 07 00:30:00.163975 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.163956 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-g89gr\"" Jul 07 00:30:00.308595 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.308564 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsc9r\" (UniqueName: \"kubernetes.io/projected/3a63e0ee-053f-41d5-b00b-4680656146ca-kube-api-access-nsc9r\") pod \"maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97\" (UID: \"3a63e0ee-053f-41d5-b00b-4680656146ca\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97" Jul 07 00:30:00.409296 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.409226 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsc9r\" (UniqueName: \"kubernetes.io/projected/3a63e0ee-053f-41d5-b00b-4680656146ca-kube-api-access-nsc9r\") pod \"maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97\" (UID: \"3a63e0ee-053f-41d5-b00b-4680656146ca\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97" Jul 07 00:30:00.417956 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.417932 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsc9r\" (UniqueName: \"kubernetes.io/projected/3a63e0ee-053f-41d5-b00b-4680656146ca-kube-api-access-nsc9r\") pod \"maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97\" (UID: \"3a63e0ee-053f-41d5-b00b-4680656146ca\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97" Jul 07 00:30:00.473221 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.473199 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97" Jul 07 00:30:00.602195 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.602132 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97"] Jul 07 00:30:00.604283 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:30:00.604255 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a63e0ee_053f_41d5_b00b_4680656146ca.slice/crio-d22494df37f696a2388efc701b95e38d05fe5b8d63c7e3ce75a54ec50ce47a11 WatchSource:0}: Error finding container d22494df37f696a2388efc701b95e38d05fe5b8d63c7e3ce75a54ec50ce47a11: Status 404 returned error can't find the container with id d22494df37f696a2388efc701b95e38d05fe5b8d63c7e3ce75a54ec50ce47a11 Jul 07 00:30:00.606242 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:00.606227 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 00:30:01.206742 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:01.206707 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97" event={"ID":"3a63e0ee-053f-41d5-b00b-4680656146ca","Type":"ContainerStarted","Data":"d22494df37f696a2388efc701b95e38d05fe5b8d63c7e3ce75a54ec50ce47a11"} Jul 07 00:30:02.615790 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:02.615756 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:30:03.184869 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.184826 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp"] Jul 07 00:30:03.185300 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.185281 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" containerName="istio-proxy" Jul 07 00:30:03.185986 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.185469 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd5a673e-e8b7-4700-a8bb-1a014c8503ea" containerName="istio-proxy" Jul 07 00:30:03.188828 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.188808 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.195180 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.195161 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-e15ad3-openshift-default-dockercfg-5ljc5\"" Jul 07 00:30:03.205047 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.205020 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp"] Jul 07 00:30:03.333407 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.333369 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-credential-socket\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.333600 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.333429 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtv9z\" (UniqueName: \"kubernetes.io/projected/8666cb23-ddfd-4e8d-a103-cf2d50c26268-kube-api-access-qtv9z\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.333600 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.333506 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-workload-socket\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.333600 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.333534 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-workload-certs\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.333600 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.333554 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-envoy\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.333600 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.333589 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-podinfo\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.333830 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.333615 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-token\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.333830 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.333650 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-data\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.333830 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.333770 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istiod-ca-cert\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.434310 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.434278 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istiod-ca-cert\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.434489 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.434317 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-credential-socket\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.434489 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.434334 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtv9z\" (UniqueName: \"kubernetes.io/projected/8666cb23-ddfd-4e8d-a103-cf2d50c26268-kube-api-access-qtv9z\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.434614 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.434488 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-workload-socket\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.434614 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.434528 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-workload-certs\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.434614 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.434562 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-envoy\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.434614 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.434597 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-podinfo\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.434818 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.434622 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-token\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.434818 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.434661 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-data\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.434818 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.434803 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-credential-socket\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.435068 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.434804 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-workload-socket\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.435068 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.434921 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-workload-certs\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.435068 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.435044 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-data\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.435210 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.435099 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istiod-ca-cert\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.436944 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.436924 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-envoy\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.437050 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.437023 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-podinfo\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.441954 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.441926 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtv9z\" (UniqueName: \"kubernetes.io/projected/8666cb23-ddfd-4e8d-a103-cf2d50c26268-kube-api-access-qtv9z\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.442050 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.441988 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-token\") pod \"e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:03.503115 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:03.503087 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:04.222214 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:04.222186 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp"] Jul 07 00:30:04.224354 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:30:04.224328 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8666cb23_ddfd_4e8d_a103_cf2d50c26268.slice/crio-858e8d6dd3f3b1b7d398262a8c732ec6259fdf7e8bfea4fd699db377ce4a1e7c WatchSource:0}: Error finding container 858e8d6dd3f3b1b7d398262a8c732ec6259fdf7e8bfea4fd699db377ce4a1e7c: Status 404 returned error can't find the container with id 858e8d6dd3f3b1b7d398262a8c732ec6259fdf7e8bfea4fd699db377ce4a1e7c Jul 07 00:30:04.226634 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:04.226594 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:30:04.226761 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:04.226661 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:30:04.226761 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:04.226699 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:30:05.233828 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:05.233797 2568 generic.go:356] "Generic (PLEG): container finished" podID="3a63e0ee-053f-41d5-b00b-4680656146ca" containerID="58755d812680cbe9718e975bbc1a1ac5c783d22a0818f5a2a341f14202da3ec2" exitCode=7 Jul 07 00:30:05.234179 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:05.233878 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97" event={"ID":"3a63e0ee-053f-41d5-b00b-4680656146ca","Type":"ContainerDied","Data":"58755d812680cbe9718e975bbc1a1ac5c783d22a0818f5a2a341f14202da3ec2"} Jul 07 00:30:05.234179 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:05.234002 2568 scope.go:117] "RemoveContainer" containerID="58755d812680cbe9718e975bbc1a1ac5c783d22a0818f5a2a341f14202da3ec2" Jul 07 00:30:05.235512 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:05.235487 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" event={"ID":"8666cb23-ddfd-4e8d-a103-cf2d50c26268","Type":"ContainerStarted","Data":"de59e987b9f5f9431baa6481300e7002f709e0c76f714aea51a9f6691be6d28b"} Jul 07 00:30:05.235619 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:05.235533 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" event={"ID":"8666cb23-ddfd-4e8d-a103-cf2d50c26268","Type":"ContainerStarted","Data":"858e8d6dd3f3b1b7d398262a8c732ec6259fdf7e8bfea4fd699db377ce4a1e7c"} Jul 07 00:30:05.298595 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:05.298556 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" podStartSLOduration=2.298542848 podStartE2EDuration="2.298542848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:30:05.296468179 +0000 UTC m=+2028.436925239" watchObservedRunningTime="2026-07-07 00:30:05.298542848 +0000 UTC m=+2028.438999902" Jul 07 00:30:05.503650 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:05.503600 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:05.508247 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:05.508218 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:06.242384 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:06.242346 2568 generic.go:356] "Generic (PLEG): container finished" podID="3a63e0ee-053f-41d5-b00b-4680656146ca" containerID="a920bd92d531d72b5ec99cc20acdc605971dce99e5af873fb470933baaebfccb" exitCode=7 Jul 07 00:30:06.242875 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:06.242441 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97" event={"ID":"3a63e0ee-053f-41d5-b00b-4680656146ca","Type":"ContainerDied","Data":"a920bd92d531d72b5ec99cc20acdc605971dce99e5af873fb470933baaebfccb"} Jul 07 00:30:06.242875 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:06.242501 2568 scope.go:117] "RemoveContainer" containerID="58755d812680cbe9718e975bbc1a1ac5c783d22a0818f5a2a341f14202da3ec2" Jul 07 00:30:06.242875 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:06.242720 2568 scope.go:117] "RemoveContainer" containerID="a920bd92d531d72b5ec99cc20acdc605971dce99e5af873fb470933baaebfccb" Jul 07 00:30:06.242875 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:06.242849 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:06.243081 ip-10-0-129-125 kubenswrapper[2568]: E0707 00:30:06.242914 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97_opendatahub(3a63e0ee-053f-41d5-b00b-4680656146ca)\"" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97" podUID="3a63e0ee-053f-41d5-b00b-4680656146ca" Jul 07 00:30:06.244107 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:06.244093 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:07.249790 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:07.249764 2568 scope.go:117] "RemoveContainer" containerID="a920bd92d531d72b5ec99cc20acdc605971dce99e5af873fb470933baaebfccb" Jul 07 00:30:07.250177 ip-10-0-129-125 kubenswrapper[2568]: E0707 00:30:07.249941 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97_opendatahub(3a63e0ee-053f-41d5-b00b-4680656146ca)\"" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97" podUID="3a63e0ee-053f-41d5-b00b-4680656146ca" Jul 07 00:30:14.621726 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:14.621686 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:30:22.285626 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:22.285594 2568 patch_prober.go:28] interesting pod/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 00:30:22.297078 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:22.285671 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" podUID="8666cb23-ddfd-4e8d-a103-cf2d50c26268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 00:30:22.480920 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:22.480885 2568 scope.go:117] "RemoveContainer" containerID="a920bd92d531d72b5ec99cc20acdc605971dce99e5af873fb470933baaebfccb" Jul 07 00:30:23.330327 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:23.330295 2568 generic.go:356] "Generic (PLEG): container finished" podID="3a63e0ee-053f-41d5-b00b-4680656146ca" containerID="fbdaebcb2347cdf5f20eb75ba6452245e45e948f227cae56644a20815631470d" exitCode=0 Jul 07 00:30:23.330803 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:23.330368 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97" event={"ID":"3a63e0ee-053f-41d5-b00b-4680656146ca","Type":"ContainerDied","Data":"fbdaebcb2347cdf5f20eb75ba6452245e45e948f227cae56644a20815631470d"} Jul 07 00:30:23.330803 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:23.330480 2568 scope.go:117] "RemoveContainer" containerID="a920bd92d531d72b5ec99cc20acdc605971dce99e5af873fb470933baaebfccb" Jul 07 00:30:23.332062 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:23.332040 2568 generic.go:356] "Generic (PLEG): container finished" podID="8666cb23-ddfd-4e8d-a103-cf2d50c26268" containerID="de59e987b9f5f9431baa6481300e7002f709e0c76f714aea51a9f6691be6d28b" exitCode=0 Jul 07 00:30:23.332158 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:23.332108 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" event={"ID":"8666cb23-ddfd-4e8d-a103-cf2d50c26268","Type":"ContainerDied","Data":"de59e987b9f5f9431baa6481300e7002f709e0c76f714aea51a9f6691be6d28b"} Jul 07 00:30:23.332497 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:23.332483 2568 scope.go:117] "RemoveContainer" containerID="de59e987b9f5f9431baa6481300e7002f709e0c76f714aea51a9f6691be6d28b" Jul 07 00:30:23.333384 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:23.333357 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:30:23.333475 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:23.333446 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:30:23.333517 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:23.333486 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 00:30:24.339769 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:24.339727 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" event={"ID":"8666cb23-ddfd-4e8d-a103-cf2d50c26268","Type":"ContainerStarted","Data":"4ae3e843cc66238974d1a472d23d84087cee8dc9655adfaed9c6efe08e2f728f"} Jul 07 00:30:24.366460 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:24.366407 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97"] Jul 07 00:30:24.503296 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:24.503265 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:24.503521 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:24.503501 2568 patch_prober.go:28] interesting pod/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 07 00:30:24.503588 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:24.503546 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" podUID="8666cb23-ddfd-4e8d-a103-cf2d50c26268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 07 00:30:24.509097 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:24.509078 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97" Jul 07 00:30:24.618371 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:24.618289 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsc9r\" (UniqueName: \"kubernetes.io/projected/3a63e0ee-053f-41d5-b00b-4680656146ca-kube-api-access-nsc9r\") pod \"3a63e0ee-053f-41d5-b00b-4680656146ca\" (UID: \"3a63e0ee-053f-41d5-b00b-4680656146ca\") " Jul 07 00:30:24.620571 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:24.620546 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a63e0ee-053f-41d5-b00b-4680656146ca-kube-api-access-nsc9r" (OuterVolumeSpecName: "kube-api-access-nsc9r") pod "3a63e0ee-053f-41d5-b00b-4680656146ca" (UID: "3a63e0ee-053f-41d5-b00b-4680656146ca"). InnerVolumeSpecName "kube-api-access-nsc9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:30:24.719111 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:24.719071 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsc9r\" (UniqueName: \"kubernetes.io/projected/3a63e0ee-053f-41d5-b00b-4680656146ca-kube-api-access-nsc9r\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:30:25.353316 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:25.353280 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97" Jul 07 00:30:25.355717 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:25.355685 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97" event={"ID":"3a63e0ee-053f-41d5-b00b-4680656146ca","Type":"ContainerDied","Data":"d22494df37f696a2388efc701b95e38d05fe5b8d63c7e3ce75a54ec50ce47a11"} Jul 07 00:30:25.355854 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:25.355739 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d22494df37f696a2388efc701b95e38d05fe5b8d63c7e3ce75a54ec50ce47a11" Jul 07 00:30:25.382878 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:25.382852 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97"] Jul 07 00:30:25.384592 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:25.384574 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-ea3965-29723070-zsj97"] Jul 07 00:30:25.486545 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:25.486510 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a63e0ee-053f-41d5-b00b-4680656146ca" path="/var/lib/kubelet/pods/3a63e0ee-053f-41d5-b00b-4680656146ca/volumes" Jul 07 00:30:25.504068 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:25.504041 2568 patch_prober.go:28] interesting pod/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 07 00:30:25.504174 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:25.504090 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" podUID="8666cb23-ddfd-4e8d-a103-cf2d50c26268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 07 00:30:26.503893 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:26.503845 2568 patch_prober.go:28] interesting pod/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 07 00:30:26.504395 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:26.503912 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" podUID="8666cb23-ddfd-4e8d-a103-cf2d50c26268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 07 00:30:27.318414 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:27.318373 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:30:27.504005 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:27.503965 2568 patch_prober.go:28] interesting pod/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 07 00:30:27.504395 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:27.504022 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" podUID="8666cb23-ddfd-4e8d-a103-cf2d50c26268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 07 00:30:28.507335 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:28.507300 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:28.507805 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:28.507654 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:28.508266 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:28.508252 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:30:56.818036 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:30:56.817955 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:31:19.018555 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:31:19.018522 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:31:36.210664 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:31:36.210633 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:31:53.120568 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:31:53.120534 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:32:10.525716 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:32:10.525633 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:32:27.636031 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:32:27.635995 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:32:46.020906 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:32:46.020872 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:33:00.616432 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:33:00.616385 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:33:17.222141 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:33:17.222109 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:33:39.014968 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:33:39.014885 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:34:00.699194 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.699163 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh"] Jul 07 00:34:00.699687 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.699468 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a63e0ee-053f-41d5-b00b-4680656146ca" containerName="cleanup" Jul 07 00:34:00.699687 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.699480 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a63e0ee-053f-41d5-b00b-4680656146ca" containerName="cleanup" Jul 07 00:34:00.699687 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.699488 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a63e0ee-053f-41d5-b00b-4680656146ca" containerName="cleanup" Jul 07 00:34:00.699687 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.699493 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a63e0ee-053f-41d5-b00b-4680656146ca" containerName="cleanup" Jul 07 00:34:00.699687 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.699501 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a63e0ee-053f-41d5-b00b-4680656146ca" containerName="cleanup" Jul 07 00:34:00.699687 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.699506 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a63e0ee-053f-41d5-b00b-4680656146ca" containerName="cleanup" Jul 07 00:34:00.699687 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.699568 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a63e0ee-053f-41d5-b00b-4680656146ca" containerName="cleanup" Jul 07 00:34:00.699687 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.699575 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a63e0ee-053f-41d5-b00b-4680656146ca" containerName="cleanup" Jul 07 00:34:00.702371 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.702357 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.705527 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.705506 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e15ad3\"/\"default-dockercfg-8bxf8\"" Jul 07 00:34:00.705647 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.705543 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e15ad3\"/\"openshift-service-ca.crt\"" Jul 07 00:34:00.705647 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.705585 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e15ad3\"/\"kube-root-ca.crt\"" Jul 07 00:34:00.705647 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.705603 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e15ad3\"/\"sub-test-model-e15ad3-kserve-self-signed-certs\"" Jul 07 00:34:00.715800 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.715780 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh"] Jul 07 00:34:00.765475 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.765454 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djzx9\" (UniqueName: \"kubernetes.io/projected/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-kube-api-access-djzx9\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.765585 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.765489 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-tls-certs\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.765585 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.765510 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-tmp-dir\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.765700 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.765617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-home\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.765700 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.765657 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-model-cache\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.765788 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.765732 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-dshm\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.866340 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.866308 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-dshm\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.866476 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.866350 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djzx9\" (UniqueName: \"kubernetes.io/projected/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-kube-api-access-djzx9\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.866476 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.866384 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-tls-certs\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.866476 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.866411 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-tmp-dir\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.866660 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.866484 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-home\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.866660 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.866515 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-model-cache\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.866866 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.866845 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-tmp-dir\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.866933 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.866864 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-home\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.866933 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.866891 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-model-cache\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.868552 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.868534 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-dshm\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.868786 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.868771 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-tls-certs\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:00.874375 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:00.874358 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djzx9\" (UniqueName: \"kubernetes.io/projected/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-kube-api-access-djzx9\") pod \"sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:01.011311 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:01.011242 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:01.137645 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:01.137617 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh"] Jul 07 00:34:01.140233 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:34:01.140206 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96d4fa04_dd9b_4f38_9a00_cbb1fab5928c.slice/crio-ce141b76681e0ba7c6ec34b0c74c2d31e44734a65516b1c4cc337b965fb8f895 WatchSource:0}: Error finding container ce141b76681e0ba7c6ec34b0c74c2d31e44734a65516b1c4cc337b965fb8f895: Status 404 returned error can't find the container with id ce141b76681e0ba7c6ec34b0c74c2d31e44734a65516b1c4cc337b965fb8f895 Jul 07 00:34:01.426923 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:01.426818 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" event={"ID":"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c","Type":"ContainerStarted","Data":"dfdab0289b01ae87666f4f6d7658244f859c9108957c15dbb0eba0365a028c04"} Jul 07 00:34:01.426923 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:01.426865 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" event={"ID":"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c","Type":"ContainerStarted","Data":"ce141b76681e0ba7c6ec34b0c74c2d31e44734a65516b1c4cc337b965fb8f895"} Jul 07 00:34:01.444544 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:01.444494 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" podStartSLOduration=1.4444795670000001 podStartE2EDuration="1.444479567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:34:01.442710011 +0000 UTC m=+2264.583167068" watchObservedRunningTime="2026-07-07 00:34:01.444479567 +0000 UTC m=+2264.584936601" Jul 07 00:34:05.312560 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:05.312529 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:34:11.011501 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:11.011470 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:11.011904 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:11.011531 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:11.024013 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:11.023989 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:11.488446 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:11.488405 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:34:31.715601 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:31.715564 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:34:51.820974 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:51.820939 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:34:58.102181 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:34:58.102149 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh"] Jul 07 00:35:10.319964 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:10.319887 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:35:13.121897 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.121834 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" podUID="96d4fa04-dd9b-4f38-9a00-cbb1fab5928c" containerName="main" containerID="cri-o://dfdab0289b01ae87666f4f6d7658244f859c9108957c15dbb0eba0365a028c04" gracePeriod=45 Jul 07 00:35:13.366881 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.366858 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:35:13.448795 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.448772 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-tmp-dir\") pod \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " Jul 07 00:35:13.448925 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.448804 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-dshm\") pod \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " Jul 07 00:35:13.448925 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.448852 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djzx9\" (UniqueName: \"kubernetes.io/projected/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-kube-api-access-djzx9\") pod \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " Jul 07 00:35:13.448925 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.448877 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-home\") pod \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " Jul 07 00:35:13.448925 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.448897 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-tls-certs\") pod \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " Jul 07 00:35:13.448925 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.448925 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-model-cache\") pod \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\" (UID: \"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c\") " Jul 07 00:35:13.449203 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.449059 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96d4fa04-dd9b-4f38-9a00-cbb1fab5928c" (UID: "96d4fa04-dd9b-4f38-9a00-cbb1fab5928c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:13.449203 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.449115 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-home" (OuterVolumeSpecName: "home") pod "96d4fa04-dd9b-4f38-9a00-cbb1fab5928c" (UID: "96d4fa04-dd9b-4f38-9a00-cbb1fab5928c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:13.449295 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.449232 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-model-cache" (OuterVolumeSpecName: "model-cache") pod "96d4fa04-dd9b-4f38-9a00-cbb1fab5928c" (UID: "96d4fa04-dd9b-4f38-9a00-cbb1fab5928c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:13.451107 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.451076 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96d4fa04-dd9b-4f38-9a00-cbb1fab5928c" (UID: "96d4fa04-dd9b-4f38-9a00-cbb1fab5928c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:35:13.451210 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.451134 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-kube-api-access-djzx9" (OuterVolumeSpecName: "kube-api-access-djzx9") pod "96d4fa04-dd9b-4f38-9a00-cbb1fab5928c" (UID: "96d4fa04-dd9b-4f38-9a00-cbb1fab5928c"). InnerVolumeSpecName "kube-api-access-djzx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:35:13.451210 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.451154 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-dshm" (OuterVolumeSpecName: "dshm") pod "96d4fa04-dd9b-4f38-9a00-cbb1fab5928c" (UID: "96d4fa04-dd9b-4f38-9a00-cbb1fab5928c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:13.550159 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.550130 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djzx9\" (UniqueName: \"kubernetes.io/projected/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-kube-api-access-djzx9\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:35:13.550159 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.550154 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:35:13.550326 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.550163 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:35:13.550326 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.550173 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:35:13.550326 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.550181 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:35:13.550326 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.550188 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:35:13.780651 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.780564 2568 generic.go:356] "Generic (PLEG): container finished" podID="96d4fa04-dd9b-4f38-9a00-cbb1fab5928c" containerID="dfdab0289b01ae87666f4f6d7658244f859c9108957c15dbb0eba0365a028c04" exitCode=0 Jul 07 00:35:13.780651 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.780622 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" Jul 07 00:35:13.780651 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.780630 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" event={"ID":"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c","Type":"ContainerDied","Data":"dfdab0289b01ae87666f4f6d7658244f859c9108957c15dbb0eba0365a028c04"} Jul 07 00:35:13.780876 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.780667 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh" event={"ID":"96d4fa04-dd9b-4f38-9a00-cbb1fab5928c","Type":"ContainerDied","Data":"ce141b76681e0ba7c6ec34b0c74c2d31e44734a65516b1c4cc337b965fb8f895"} Jul 07 00:35:13.780876 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.780689 2568 scope.go:117] "RemoveContainer" containerID="dfdab0289b01ae87666f4f6d7658244f859c9108957c15dbb0eba0365a028c04" Jul 07 00:35:13.789100 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.789083 2568 scope.go:117] "RemoveContainer" containerID="dfdab0289b01ae87666f4f6d7658244f859c9108957c15dbb0eba0365a028c04" Jul 07 00:35:13.789330 ip-10-0-129-125 kubenswrapper[2568]: E0707 00:35:13.789313 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfdab0289b01ae87666f4f6d7658244f859c9108957c15dbb0eba0365a028c04\": container with ID starting with dfdab0289b01ae87666f4f6d7658244f859c9108957c15dbb0eba0365a028c04 not found: ID does not exist" containerID="dfdab0289b01ae87666f4f6d7658244f859c9108957c15dbb0eba0365a028c04" Jul 07 00:35:13.789431 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.789337 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfdab0289b01ae87666f4f6d7658244f859c9108957c15dbb0eba0365a028c04"} err="failed to get container status \"dfdab0289b01ae87666f4f6d7658244f859c9108957c15dbb0eba0365a028c04\": rpc error: code = NotFound desc = could not find container \"dfdab0289b01ae87666f4f6d7658244f859c9108957c15dbb0eba0365a028c04\": container with ID starting with dfdab0289b01ae87666f4f6d7658244f859c9108957c15dbb0eba0365a028c04 not found: ID does not exist" Jul 07 00:35:13.797239 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.797214 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh"] Jul 07 00:35:13.802480 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:13.802460 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e15ad3/sub-test-model-e15ad3-kserve-58ff8bff57-2x2xh"] Jul 07 00:35:15.486079 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:15.486046 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96d4fa04-dd9b-4f38-9a00-cbb1fab5928c" path="/var/lib/kubelet/pods/96d4fa04-dd9b-4f38-9a00-cbb1fab5928c/volumes" Jul 07 00:35:28.819791 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:28.819753 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:35:43.614187 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:43.614151 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:35:59.816905 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:35:59.816871 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:36:08.172391 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:08.172358 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7404-139911069003088"] Jul 07 00:36:08.172775 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:08.172661 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d4fa04-dd9b-4f38-9a00-cbb1fab5928c" containerName="main" Jul 07 00:36:08.172775 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:08.172672 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d4fa04-dd9b-4f38-9a00-cbb1fab5928c" containerName="main" Jul 07 00:36:08.172775 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:08.172732 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d4fa04-dd9b-4f38-9a00-cbb1fab5928c" containerName="main" Jul 07 00:36:08.172775 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:08.172743 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a63e0ee-053f-41d5-b00b-4680656146ca" containerName="cleanup" Jul 07 00:36:08.175537 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:08.175521 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7404-139911069003088" Jul 07 00:36:08.178264 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:08.178239 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vrnjc\"" Jul 07 00:36:08.183249 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:08.183224 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7404-139911069003088"] Jul 07 00:36:08.289759 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:08.289725 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr6v8\" (UniqueName: \"kubernetes.io/projected/54cc61b4-8a8c-46d1-a0b1-723ac5bd1715-kube-api-access-qr6v8\") pod \"test-curl-7404-139911069003088\" (UID: \"54cc61b4-8a8c-46d1-a0b1-723ac5bd1715\") " pod="opendatahub/test-curl-7404-139911069003088" Jul 07 00:36:08.390944 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:08.390916 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr6v8\" (UniqueName: \"kubernetes.io/projected/54cc61b4-8a8c-46d1-a0b1-723ac5bd1715-kube-api-access-qr6v8\") pod \"test-curl-7404-139911069003088\" (UID: \"54cc61b4-8a8c-46d1-a0b1-723ac5bd1715\") " pod="opendatahub/test-curl-7404-139911069003088" Jul 07 00:36:08.399127 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:08.399104 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr6v8\" (UniqueName: \"kubernetes.io/projected/54cc61b4-8a8c-46d1-a0b1-723ac5bd1715-kube-api-access-qr6v8\") pod \"test-curl-7404-139911069003088\" (UID: \"54cc61b4-8a8c-46d1-a0b1-723ac5bd1715\") " pod="opendatahub/test-curl-7404-139911069003088" Jul 07 00:36:08.490106 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:08.490083 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7404-139911069003088" Jul 07 00:36:08.615276 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:08.615251 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7404-139911069003088"] Jul 07 00:36:08.617252 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:36:08.617224 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54cc61b4_8a8c_46d1_a0b1_723ac5bd1715.slice/crio-6aa8820b6e60f32e8df3e6c9cc6b2d0c0f9e34733aa69e72fe43a0d0d17b7a53 WatchSource:0}: Error finding container 6aa8820b6e60f32e8df3e6c9cc6b2d0c0f9e34733aa69e72fe43a0d0d17b7a53: Status 404 returned error can't find the container with id 6aa8820b6e60f32e8df3e6c9cc6b2d0c0f9e34733aa69e72fe43a0d0d17b7a53 Jul 07 00:36:08.618524 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:08.618507 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 00:36:09.046386 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:09.046346 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7404-139911069003088" event={"ID":"54cc61b4-8a8c-46d1-a0b1-723ac5bd1715","Type":"ContainerStarted","Data":"6aa8820b6e60f32e8df3e6c9cc6b2d0c0f9e34733aa69e72fe43a0d0d17b7a53"} Jul 07 00:36:10.052544 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:10.052512 2568 generic.go:356] "Generic (PLEG): container finished" podID="54cc61b4-8a8c-46d1-a0b1-723ac5bd1715" containerID="5116c8747bb1a46ac92e82def848d3deac1295b6d64297bd770addf54f015d06" exitCode=0 Jul 07 00:36:10.052912 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:10.052551 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7404-139911069003088" event={"ID":"54cc61b4-8a8c-46d1-a0b1-723ac5bd1715","Type":"ContainerDied","Data":"5116c8747bb1a46ac92e82def848d3deac1295b6d64297bd770addf54f015d06"} Jul 07 00:36:11.189390 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.189369 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7404-139911069003088" Jul 07 00:36:11.221279 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.221255 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7404-139911069003088_54cc61b4-8a8c-46d1-a0b1-723ac5bd1715/test-curl-7404-139911069003088/0.log" Jul 07 00:36:11.242943 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.242922 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7404-139911069003088"] Jul 07 00:36:11.244772 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.244755 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7404-139911069003088"] Jul 07 00:36:11.314023 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.313963 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr6v8\" (UniqueName: \"kubernetes.io/projected/54cc61b4-8a8c-46d1-a0b1-723ac5bd1715-kube-api-access-qr6v8\") pod \"54cc61b4-8a8c-46d1-a0b1-723ac5bd1715\" (UID: \"54cc61b4-8a8c-46d1-a0b1-723ac5bd1715\") " Jul 07 00:36:11.315972 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.315948 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54cc61b4-8a8c-46d1-a0b1-723ac5bd1715-kube-api-access-qr6v8" (OuterVolumeSpecName: "kube-api-access-qr6v8") pod "54cc61b4-8a8c-46d1-a0b1-723ac5bd1715" (UID: "54cc61b4-8a8c-46d1-a0b1-723ac5bd1715"). InnerVolumeSpecName "kube-api-access-qr6v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:36:11.374770 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.374746 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7404-139911069003200"] Jul 07 00:36:11.375041 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.375029 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54cc61b4-8a8c-46d1-a0b1-723ac5bd1715" containerName="test-curl-7404-139911069003088" Jul 07 00:36:11.375089 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.375043 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="54cc61b4-8a8c-46d1-a0b1-723ac5bd1715" containerName="test-curl-7404-139911069003088" Jul 07 00:36:11.375124 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.375100 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="54cc61b4-8a8c-46d1-a0b1-723ac5bd1715" containerName="test-curl-7404-139911069003088" Jul 07 00:36:11.377885 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.377871 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7404-139911069003200" Jul 07 00:36:11.384262 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.384164 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7404-139911069003200"] Jul 07 00:36:11.414632 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.414606 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr6v8\" (UniqueName: \"kubernetes.io/projected/54cc61b4-8a8c-46d1-a0b1-723ac5bd1715-kube-api-access-qr6v8\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:36:11.485864 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.485844 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54cc61b4-8a8c-46d1-a0b1-723ac5bd1715" path="/var/lib/kubelet/pods/54cc61b4-8a8c-46d1-a0b1-723ac5bd1715/volumes" Jul 07 00:36:11.515688 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.515664 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmm7\" (UniqueName: \"kubernetes.io/projected/f3313233-f4fd-4b4a-a23b-45b880f1f277-kube-api-access-kbmm7\") pod \"test-curl-7404-139911069003200\" (UID: \"f3313233-f4fd-4b4a-a23b-45b880f1f277\") " pod="opendatahub/test-curl-7404-139911069003200" Jul 07 00:36:11.617006 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.616954 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmm7\" (UniqueName: \"kubernetes.io/projected/f3313233-f4fd-4b4a-a23b-45b880f1f277-kube-api-access-kbmm7\") pod \"test-curl-7404-139911069003200\" (UID: \"f3313233-f4fd-4b4a-a23b-45b880f1f277\") " pod="opendatahub/test-curl-7404-139911069003200" Jul 07 00:36:11.624791 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.624767 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmm7\" (UniqueName: \"kubernetes.io/projected/f3313233-f4fd-4b4a-a23b-45b880f1f277-kube-api-access-kbmm7\") pod \"test-curl-7404-139911069003200\" (UID: \"f3313233-f4fd-4b4a-a23b-45b880f1f277\") " pod="opendatahub/test-curl-7404-139911069003200" Jul 07 00:36:11.687380 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.687357 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7404-139911069003200" Jul 07 00:36:11.812346 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:11.812323 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7404-139911069003200"] Jul 07 00:36:11.814622 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:36:11.814598 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3313233_f4fd_4b4a_a23b_45b880f1f277.slice/crio-17ea1568c0e20adbe8dfc47ad0d95d37f07729dfb8826a7dd271bd1d69b6d1f5 WatchSource:0}: Error finding container 17ea1568c0e20adbe8dfc47ad0d95d37f07729dfb8826a7dd271bd1d69b6d1f5: Status 404 returned error can't find the container with id 17ea1568c0e20adbe8dfc47ad0d95d37f07729dfb8826a7dd271bd1d69b6d1f5 Jul 07 00:36:12.063725 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:12.063700 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7404-139911069003088" Jul 07 00:36:12.063880 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:12.063749 2568 scope.go:117] "RemoveContainer" containerID="5116c8747bb1a46ac92e82def848d3deac1295b6d64297bd770addf54f015d06" Jul 07 00:36:12.065027 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:12.065001 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7404-139911069003200" event={"ID":"f3313233-f4fd-4b4a-a23b-45b880f1f277","Type":"ContainerStarted","Data":"17ea1568c0e20adbe8dfc47ad0d95d37f07729dfb8826a7dd271bd1d69b6d1f5"} Jul 07 00:36:13.072748 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:13.072713 2568 generic.go:356] "Generic (PLEG): container finished" podID="f3313233-f4fd-4b4a-a23b-45b880f1f277" containerID="2aee5da05e7fba8b309b5835af4aad5fc2dbb794da5b8d2616793bab5fa720e6" exitCode=0 Jul 07 00:36:13.073147 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:13.072782 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7404-139911069003200" event={"ID":"f3313233-f4fd-4b4a-a23b-45b880f1f277","Type":"ContainerDied","Data":"2aee5da05e7fba8b309b5835af4aad5fc2dbb794da5b8d2616793bab5fa720e6"} Jul 07 00:36:13.720543 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:13.720511 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:36:14.222771 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:14.222745 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7404-139911069003200" Jul 07 00:36:14.241475 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:14.241446 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7404-139911069003200_f3313233-f4fd-4b4a-a23b-45b880f1f277/test-curl-7404-139911069003200/0.log" Jul 07 00:36:14.264084 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:14.264055 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7404-139911069003200"] Jul 07 00:36:14.268227 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:14.268202 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7404-139911069003200"] Jul 07 00:36:14.338319 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:14.338255 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbmm7\" (UniqueName: \"kubernetes.io/projected/f3313233-f4fd-4b4a-a23b-45b880f1f277-kube-api-access-kbmm7\") pod \"f3313233-f4fd-4b4a-a23b-45b880f1f277\" (UID: \"f3313233-f4fd-4b4a-a23b-45b880f1f277\") " Jul 07 00:36:14.340219 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:14.340201 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3313233-f4fd-4b4a-a23b-45b880f1f277-kube-api-access-kbmm7" (OuterVolumeSpecName: "kube-api-access-kbmm7") pod "f3313233-f4fd-4b4a-a23b-45b880f1f277" (UID: "f3313233-f4fd-4b4a-a23b-45b880f1f277"). InnerVolumeSpecName "kube-api-access-kbmm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:36:14.439711 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:14.439680 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbmm7\" (UniqueName: \"kubernetes.io/projected/f3313233-f4fd-4b4a-a23b-45b880f1f277-kube-api-access-kbmm7\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:36:15.083789 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:15.083762 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7404-139911069003200" Jul 07 00:36:15.083969 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:15.083792 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17ea1568c0e20adbe8dfc47ad0d95d37f07729dfb8826a7dd271bd1d69b6d1f5" Jul 07 00:36:15.093036 ip-10-0-129-125 kubenswrapper[2568]: E0707 00:36:15.093006 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7404-139911069003200\" is forbidden: User \"system:node:ip-10-0-129-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.ec2.internal' and this object" podUID="f3313233-f4fd-4b4a-a23b-45b880f1f277" pod="opendatahub/test-curl-7404-139911069003200" Jul 07 00:36:15.485566 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:15.485538 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3313233-f4fd-4b4a-a23b-45b880f1f277" path="/var/lib/kubelet/pods/f3313233-f4fd-4b4a-a23b-45b880f1f277/volumes" Jul 07 00:36:54.918386 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:36:54.918305 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:37:04.602182 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:04.602143 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp"] Jul 07 00:37:04.602689 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:04.602385 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" podUID="8666cb23-ddfd-4e8d-a103-cf2d50c26268" containerName="istio-proxy" containerID="cri-o://4ae3e843cc66238974d1a472d23d84087cee8dc9655adfaed9c6efe08e2f728f" gracePeriod=30 Jul 07 00:37:07.518211 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:07.518180 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4c2k5"] Jul 07 00:37:09.859524 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.859495 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:37:09.877814 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.877737 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-data\") pod \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " Jul 07 00:37:09.877814 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.877799 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-workload-socket\") pod \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " Jul 07 00:37:09.878006 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.877837 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-workload-certs\") pod \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " Jul 07 00:37:09.878006 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.877859 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-token\") pod \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " Jul 07 00:37:09.878006 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.877881 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istiod-ca-cert\") pod \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " Jul 07 00:37:09.878006 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.877910 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-podinfo\") pod \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " Jul 07 00:37:09.878006 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.877951 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtv9z\" (UniqueName: \"kubernetes.io/projected/8666cb23-ddfd-4e8d-a103-cf2d50c26268-kube-api-access-qtv9z\") pod \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " Jul 07 00:37:09.878006 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.877996 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-credential-socket\") pod \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " Jul 07 00:37:09.878297 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.878020 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-envoy\") pod \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\" (UID: \"8666cb23-ddfd-4e8d-a103-cf2d50c26268\") " Jul 07 00:37:09.878297 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.878122 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8666cb23-ddfd-4e8d-a103-cf2d50c26268" (UID: "8666cb23-ddfd-4e8d-a103-cf2d50c26268"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:37:09.878297 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.878274 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-workload-socket\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:37:09.878787 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.878479 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8666cb23-ddfd-4e8d-a103-cf2d50c26268" (UID: "8666cb23-ddfd-4e8d-a103-cf2d50c26268"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:37:09.878787 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.878638 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8666cb23-ddfd-4e8d-a103-cf2d50c26268" (UID: "8666cb23-ddfd-4e8d-a103-cf2d50c26268"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:37:09.878954 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.878798 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8666cb23-ddfd-4e8d-a103-cf2d50c26268" (UID: "8666cb23-ddfd-4e8d-a103-cf2d50c26268"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:37:09.878954 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.878833 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-data" (OuterVolumeSpecName: "istio-data") pod "8666cb23-ddfd-4e8d-a103-cf2d50c26268" (UID: "8666cb23-ddfd-4e8d-a103-cf2d50c26268"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:37:09.880700 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.880674 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-token" (OuterVolumeSpecName: "istio-token") pod "8666cb23-ddfd-4e8d-a103-cf2d50c26268" (UID: "8666cb23-ddfd-4e8d-a103-cf2d50c26268"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:37:09.880809 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.880706 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8666cb23-ddfd-4e8d-a103-cf2d50c26268-kube-api-access-qtv9z" (OuterVolumeSpecName: "kube-api-access-qtv9z") pod "8666cb23-ddfd-4e8d-a103-cf2d50c26268" (UID: "8666cb23-ddfd-4e8d-a103-cf2d50c26268"). InnerVolumeSpecName "kube-api-access-qtv9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:37:09.880927 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.880912 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8666cb23-ddfd-4e8d-a103-cf2d50c26268" (UID: "8666cb23-ddfd-4e8d-a103-cf2d50c26268"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:37:09.881043 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.881025 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8666cb23-ddfd-4e8d-a103-cf2d50c26268" (UID: "8666cb23-ddfd-4e8d-a103-cf2d50c26268"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:37:09.979687 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.979651 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-data\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:37:09.979687 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.979680 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-workload-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:37:09.979687 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.979689 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-token\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:37:09.979687 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.979696 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istiod-ca-cert\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:37:09.979958 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.979704 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-podinfo\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:37:09.979958 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.979712 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtv9z\" (UniqueName: \"kubernetes.io/projected/8666cb23-ddfd-4e8d-a103-cf2d50c26268-kube-api-access-qtv9z\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:37:09.979958 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.979722 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-credential-socket\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:37:09.979958 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:09.979730 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8666cb23-ddfd-4e8d-a103-cf2d50c26268-istio-envoy\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\"" Jul 07 00:37:10.354917 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:10.354880 2568 generic.go:356] "Generic (PLEG): container finished" podID="8666cb23-ddfd-4e8d-a103-cf2d50c26268" containerID="4ae3e843cc66238974d1a472d23d84087cee8dc9655adfaed9c6efe08e2f728f" exitCode=0 Jul 07 00:37:10.355103 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:10.354924 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" event={"ID":"8666cb23-ddfd-4e8d-a103-cf2d50c26268","Type":"ContainerDied","Data":"4ae3e843cc66238974d1a472d23d84087cee8dc9655adfaed9c6efe08e2f728f"} Jul 07 00:37:10.355103 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:10.354939 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" Jul 07 00:37:10.355103 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:10.354975 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp" event={"ID":"8666cb23-ddfd-4e8d-a103-cf2d50c26268","Type":"ContainerDied","Data":"858e8d6dd3f3b1b7d398262a8c732ec6259fdf7e8bfea4fd699db377ce4a1e7c"} Jul 07 00:37:10.355103 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:10.355000 2568 scope.go:117] "RemoveContainer" containerID="4ae3e843cc66238974d1a472d23d84087cee8dc9655adfaed9c6efe08e2f728f" Jul 07 00:37:10.364574 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:10.364556 2568 scope.go:117] "RemoveContainer" containerID="de59e987b9f5f9431baa6481300e7002f709e0c76f714aea51a9f6691be6d28b" Jul 07 00:37:10.372275 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:10.372255 2568 scope.go:117] "RemoveContainer" containerID="4ae3e843cc66238974d1a472d23d84087cee8dc9655adfaed9c6efe08e2f728f" Jul 07 00:37:10.372528 ip-10-0-129-125 kubenswrapper[2568]: E0707 00:37:10.372507 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae3e843cc66238974d1a472d23d84087cee8dc9655adfaed9c6efe08e2f728f\": container with ID starting with 4ae3e843cc66238974d1a472d23d84087cee8dc9655adfaed9c6efe08e2f728f not found: ID does not exist" containerID="4ae3e843cc66238974d1a472d23d84087cee8dc9655adfaed9c6efe08e2f728f" Jul 07 00:37:10.372601 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:10.372541 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae3e843cc66238974d1a472d23d84087cee8dc9655adfaed9c6efe08e2f728f"} err="failed to get container status \"4ae3e843cc66238974d1a472d23d84087cee8dc9655adfaed9c6efe08e2f728f\": rpc error: code = NotFound desc = could not find container \"4ae3e843cc66238974d1a472d23d84087cee8dc9655adfaed9c6efe08e2f728f\": container with ID starting with 4ae3e843cc66238974d1a472d23d84087cee8dc9655adfaed9c6efe08e2f728f not found: ID does not exist" Jul 07 00:37:10.372601 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:10.372565 2568 scope.go:117] "RemoveContainer" containerID="de59e987b9f5f9431baa6481300e7002f709e0c76f714aea51a9f6691be6d28b" Jul 07 00:37:10.372838 ip-10-0-129-125 kubenswrapper[2568]: E0707 00:37:10.372821 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de59e987b9f5f9431baa6481300e7002f709e0c76f714aea51a9f6691be6d28b\": container with ID starting with de59e987b9f5f9431baa6481300e7002f709e0c76f714aea51a9f6691be6d28b not found: ID does not exist" containerID="de59e987b9f5f9431baa6481300e7002f709e0c76f714aea51a9f6691be6d28b" Jul 07 00:37:10.372887 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:10.372844 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de59e987b9f5f9431baa6481300e7002f709e0c76f714aea51a9f6691be6d28b"} err="failed to get container status \"de59e987b9f5f9431baa6481300e7002f709e0c76f714aea51a9f6691be6d28b\": rpc error: code = NotFound desc = could not find container \"de59e987b9f5f9431baa6481300e7002f709e0c76f714aea51a9f6691be6d28b\": container with ID starting with de59e987b9f5f9431baa6481300e7002f709e0c76f714aea51a9f6691be6d28b not found: ID does not exist" Jul 07 00:37:10.377074 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:10.377048 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp"] Jul 07 00:37:10.380580 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:10.380559 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-e15ad3-openshift-default-548bf477f9-blrqp"] Jul 07 00:37:11.299063 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:11.299020 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fcdfc77c6-wqnxm_7091db00-88eb-40b5-91a1-6ca17af6a084/manager/0.log" Jul 07 00:37:11.304995 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:11.304976 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fcdfc77c6-wqnxm_7091db00-88eb-40b5-91a1-6ca17af6a084/copy-manifests/0.log" Jul 07 00:37:11.486205 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:11.486174 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8666cb23-ddfd-4e8d-a103-cf2d50c26268" path="/var/lib/kubelet/pods/8666cb23-ddfd-4e8d-a103-cf2d50c26268/volumes" Jul 07 00:37:12.196469 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:12.196440 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5_93feb81c-29f0-48d8-8001-f00293c545dc/util/0.log" Jul 07 00:37:12.203293 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:12.203268 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5_93feb81c-29f0-48d8-8001-f00293c545dc/pull/0.log" Jul 07 00:37:12.210649 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:12.210627 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5_93feb81c-29f0-48d8-8001-f00293c545dc/extract/0.log" Jul 07 00:37:12.331651 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:12.331625 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl_c1a71ff1-2865-4800-8b89-8c62a56ee213/util/0.log" Jul 07 00:37:12.338327 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:12.338303 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl_c1a71ff1-2865-4800-8b89-8c62a56ee213/pull/0.log" Jul 07 00:37:12.344816 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:12.344794 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl_c1a71ff1-2865-4800-8b89-8c62a56ee213/extract/0.log" Jul 07 00:37:12.461843 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:12.461751 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n_0582860a-f733-4781-a75b-17f86eb6447c/util/0.log" Jul 07 00:37:12.468118 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:12.468086 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n_0582860a-f733-4781-a75b-17f86eb6447c/pull/0.log" Jul 07 00:37:12.475298 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:12.475277 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n_0582860a-f733-4781-a75b-17f86eb6447c/extract/0.log" Jul 07 00:37:12.603744 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:12.603717 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl_9872f2e0-1bdc-41d4-bd52-250eb055a764/util/0.log" Jul 07 00:37:12.615905 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:12.615880 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl_9872f2e0-1bdc-41d4-bd52-250eb055a764/pull/0.log" Jul 07 00:37:12.626664 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:12.626640 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl_9872f2e0-1bdc-41d4-bd52-250eb055a764/extract/0.log" Jul 07 00:37:13.507632 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:13.507607 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-4c2k5_eb2d7414-4503-4b22-ac1b-51fa9c412719/limitador/0.log" Jul 07 00:37:14.116829 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:14.116794 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9br2p_9c75d750-3b58-42ff-9951-d40d64c8c75f/discovery/0.log" Jul 07 00:37:15.476869 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:15.476839 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-k4wkq_6280ca7c-b127-45b1-9b6b-769069baa5f5/main/0.log" Jul 07 00:37:17.655544 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:17.655514 2568 scope.go:117] "RemoveContainer" containerID="fbdaebcb2347cdf5f20eb75ba6452245e45e948f227cae56644a20815631470d" Jul 07 00:37:23.165615 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:23.165584 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x5k59_76bea87f-a057-4a59-8e4a-171542fa018a/global-pull-secret-syncer/0.log" Jul 07 00:37:23.302692 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:23.302668 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tdb62_38be6bda-4bd1-4d2f-912f-2c1f0b372ed0/konnectivity-agent/0.log" Jul 07 00:37:23.323611 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:23.323587 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-125.ec2.internal_2fa7c948617b9fac2532d316a93a8386/haproxy/0.log" Jul 07 00:37:28.005209 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:28.005181 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5_93feb81c-29f0-48d8-8001-f00293c545dc/extract/0.log" Jul 07 00:37:28.027562 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:28.027537 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5_93feb81c-29f0-48d8-8001-f00293c545dc/util/0.log" Jul 07 00:37:28.054064 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:28.054033 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bjrv5_93feb81c-29f0-48d8-8001-f00293c545dc/pull/0.log" Jul 07 00:37:28.079899 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:28.079872 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl_c1a71ff1-2865-4800-8b89-8c62a56ee213/extract/0.log" Jul 07 00:37:28.099580 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:28.099559 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl_c1a71ff1-2865-4800-8b89-8c62a56ee213/util/0.log" Jul 07 00:37:28.119905 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:28.119882 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j6shl_c1a71ff1-2865-4800-8b89-8c62a56ee213/pull/0.log" Jul 07 00:37:28.145336 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:28.145319 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n_0582860a-f733-4781-a75b-17f86eb6447c/extract/0.log" Jul 07 00:37:28.167616 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:28.167601 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n_0582860a-f733-4781-a75b-17f86eb6447c/util/0.log" Jul 07 00:37:28.188760 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:28.188740 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vkc2n_0582860a-f733-4781-a75b-17f86eb6447c/pull/0.log" Jul 07 00:37:28.221220 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:28.221200 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl_9872f2e0-1bdc-41d4-bd52-250eb055a764/extract/0.log" Jul 07 00:37:28.243266 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:28.243248 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl_9872f2e0-1bdc-41d4-bd52-250eb055a764/util/0.log" Jul 07 00:37:28.269063 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:28.269010 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1c5fwl_9872f2e0-1bdc-41d4-bd52-250eb055a764/pull/0.log" Jul 07 00:37:29.146362 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:29.146328 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-4c2k5_eb2d7414-4503-4b22-ac1b-51fa9c412719/limitador/0.log" Jul 07 00:37:31.390223 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:31.390192 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xsr4v_ccb34dac-83d6-4197-88ce-b709d828ff52/node-exporter/0.log" Jul 07 00:37:31.418552 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:31.418533 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xsr4v_ccb34dac-83d6-4197-88ce-b709d828ff52/kube-rbac-proxy/0.log" Jul 07 00:37:31.440252 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:31.440231 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xsr4v_ccb34dac-83d6-4197-88ce-b709d828ff52/init-textfile/0.log" Jul 07 00:37:31.846805 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:31.846774 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-dntg6_ca82b883-02f0-4aeb-a8b9-4b0a07b4069f/prometheus-operator-admission-webhook/0.log" Jul 07 00:37:33.247224 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.247192 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz"] Jul 07 00:37:33.247622 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.247491 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3313233-f4fd-4b4a-a23b-45b880f1f277" containerName="test-curl-7404-139911069003200" Jul 07 00:37:33.247622 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.247503 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3313233-f4fd-4b4a-a23b-45b880f1f277" containerName="test-curl-7404-139911069003200" Jul 07 00:37:33.247622 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.247523 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8666cb23-ddfd-4e8d-a103-cf2d50c26268" containerName="istio-proxy" Jul 07 00:37:33.247622 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.247529 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8666cb23-ddfd-4e8d-a103-cf2d50c26268" containerName="istio-proxy" Jul 07 00:37:33.247622 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.247538 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8666cb23-ddfd-4e8d-a103-cf2d50c26268" containerName="istio-proxy" Jul 07 00:37:33.247622 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.247543 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8666cb23-ddfd-4e8d-a103-cf2d50c26268" containerName="istio-proxy" Jul 07 00:37:33.247622 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.247594 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3313233-f4fd-4b4a-a23b-45b880f1f277" containerName="test-curl-7404-139911069003200" Jul 07 00:37:33.247622 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.247601 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8666cb23-ddfd-4e8d-a103-cf2d50c26268" containerName="istio-proxy" Jul 07 00:37:33.250598 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.250575 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.251322 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.251294 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-xxxjj_67bc5dc2-928f-4a3c-b3df-5b0f0fb1d7ee/networking-console-plugin/0.log" Jul 07 00:37:33.252925 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.252908 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vhskf\"/\"kube-root-ca.crt\"" Jul 07 00:37:33.253012 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.252992 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vhskf\"/\"openshift-service-ca.crt\"" Jul 07 00:37:33.253791 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.253772 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vhskf\"/\"default-dockercfg-hdd62\"" Jul 07 00:37:33.262905 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.262885 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz"] Jul 07 00:37:33.355647 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.355621 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xk74\" (UniqueName: \"kubernetes.io/projected/e11a4674-757e-49b1-bcaf-65b25174aee1-kube-api-access-2xk74\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.355745 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.355670 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e11a4674-757e-49b1-bcaf-65b25174aee1-podres\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.355745 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.355717 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e11a4674-757e-49b1-bcaf-65b25174aee1-lib-modules\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.355846 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.355764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e11a4674-757e-49b1-bcaf-65b25174aee1-proc\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.355846 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.355789 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e11a4674-757e-49b1-bcaf-65b25174aee1-sys\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.456913 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.456884 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xk74\" (UniqueName: \"kubernetes.io/projected/e11a4674-757e-49b1-bcaf-65b25174aee1-kube-api-access-2xk74\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.457029 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.456925 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e11a4674-757e-49b1-bcaf-65b25174aee1-podres\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.457029 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.456945 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e11a4674-757e-49b1-bcaf-65b25174aee1-lib-modules\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.457029 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.456966 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e11a4674-757e-49b1-bcaf-65b25174aee1-proc\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.457029 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.456997 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e11a4674-757e-49b1-bcaf-65b25174aee1-sys\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.457241 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.457081 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e11a4674-757e-49b1-bcaf-65b25174aee1-lib-modules\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.457241 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.457087 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e11a4674-757e-49b1-bcaf-65b25174aee1-podres\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.457241 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.457081 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e11a4674-757e-49b1-bcaf-65b25174aee1-sys\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.457241 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.457102 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e11a4674-757e-49b1-bcaf-65b25174aee1-proc\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.464942 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.464920 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xk74\" (UniqueName: \"kubernetes.io/projected/e11a4674-757e-49b1-bcaf-65b25174aee1-kube-api-access-2xk74\") pod \"perf-node-gather-daemonset-vcrlz\" (UID: \"e11a4674-757e-49b1-bcaf-65b25174aee1\") " pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.560328 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.560272 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:33.688102 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:33.688078 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz"] Jul 07 00:37:33.690699 ip-10-0-129-125 kubenswrapper[2568]: W0707 00:37:33.690671 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode11a4674_757e_49b1_bcaf_65b25174aee1.slice/crio-6dbe4f23588a452aa219da2c7ec9569367a94c76ccfa153ef6e1d36581db9796 WatchSource:0}: Error finding container 6dbe4f23588a452aa219da2c7ec9569367a94c76ccfa153ef6e1d36581db9796: Status 404 returned error can't find the container with id 6dbe4f23588a452aa219da2c7ec9569367a94c76ccfa153ef6e1d36581db9796 Jul 07 00:37:34.339015 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:34.338983 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d8b5fc77c-4bxrq_361d91ec-77f2-4f7f-b082-f0718ebf420c/console/0.log" Jul 07 00:37:34.376561 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:34.376537 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-76gvs_6f4ba1c0-2197-47b3-872c-564ecba6c44e/download-server/0.log" Jul 07 00:37:34.471380 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:34.471345 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" event={"ID":"e11a4674-757e-49b1-bcaf-65b25174aee1","Type":"ContainerStarted","Data":"4d8f6e8356d54cdc860b8c174b97a368af57da3b49afa0bc738dde952fa73f65"} Jul 07 00:37:34.471516 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:34.471396 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" event={"ID":"e11a4674-757e-49b1-bcaf-65b25174aee1","Type":"ContainerStarted","Data":"6dbe4f23588a452aa219da2c7ec9569367a94c76ccfa153ef6e1d36581db9796"} Jul 07 00:37:34.471516 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:34.471446 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:34.487319 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:34.487281 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" podStartSLOduration=1.487267277 podStartE2EDuration="1.487267277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 00:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:37:34.486010058 +0000 UTC m=+2477.626467127" watchObservedRunningTime="2026-07-07 00:37:34.487267277 +0000 UTC m=+2477.627724311" Jul 07 00:37:35.929083 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:35.929042 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4txr6_6e160535-0c07-49d0-b690-18ca6aee0d87/dns/0.log" Jul 07 00:37:35.949402 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:35.949373 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4txr6_6e160535-0c07-49d0-b690-18ca6aee0d87/kube-rbac-proxy/0.log" Jul 07 00:37:36.112757 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:36.112724 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s8cmz_e3979fa5-6b05-4e90-abf0-ae3cb0286f0c/dns-node-resolver/0.log" Jul 07 00:37:36.610111 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:36.610071 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29723040-g55hm_27a6a626-55a6-4b1d-bca6-5377c676aae1/image-pruner/0.log" Jul 07 00:37:36.660111 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:36.660083 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5dd6b96ffb-hq86m_73c6558c-cd86-406f-940a-fae3c81fa41a/registry/0.log" Jul 07 00:37:36.683004 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:36.682977 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4t5gn_56059632-3851-4ea0-8a86-fcabfc509e65/node-ca/0.log" Jul 07 00:37:37.860582 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:37.860554 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9br2p_9c75d750-3b58-42ff-9951-d40d64c8c75f/discovery/0.log" Jul 07 00:37:38.873056 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:38.873025 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fmpf9_b80472f8-19c1-4b5b-b721-b611a82cc28c/serve-healthcheck-canary/0.log" Jul 07 00:37:39.391234 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:39.391207 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bbg4s_47bf1021-e8fc-4b54-b1b3-de741519d372/kube-rbac-proxy/0.log" Jul 07 00:37:39.412454 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:39.412408 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bbg4s_47bf1021-e8fc-4b54-b1b3-de741519d372/exporter/0.log" Jul 07 00:37:39.433877 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:39.433856 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bbg4s_47bf1021-e8fc-4b54-b1b3-de741519d372/extractor/0.log" Jul 07 00:37:40.485503 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:40.485480 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vhskf/perf-node-gather-daemonset-vcrlz" Jul 07 00:37:42.611744 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:42.611716 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fcdfc77c6-wqnxm_7091db00-88eb-40b5-91a1-6ca17af6a084/manager/0.log" Jul 07 00:37:42.633511 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:42.633487 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fcdfc77c6-wqnxm_7091db00-88eb-40b5-91a1-6ca17af6a084/copy-manifests/0.log" Jul 07 00:37:44.020007 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:44.019982 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9br2p_9c75d750-3b58-42ff-9951-d40d64c8c75f/discovery/0.log" Jul 07 00:37:44.067721 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:44.067697 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c45554c7d-zplz4_9a2aeb0b-b009-46bf-bf6b-4013269603b6/manager/0.log" Jul 07 00:37:44.097071 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:44.097045 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-6pd4f_6a94c09d-7726-40ba-8c4f-af646ba02179/openshift-lws-operator/0.log" Jul 07 00:37:45.628635 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:45.628604 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9br2p_9c75d750-3b58-42ff-9951-d40d64c8c75f/discovery/0.log" Jul 07 00:37:48.926464 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:48.926361 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-2nmxh_3c348d75-7127-444e-9da4-3a8845d0a8a9/migrator/0.log" Jul 07 00:37:48.947763 ip-10-0-129-125 kubenswrapper[2568]: I0707 00:37:48.947741 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-2nmxh_3c348d75-7127-444e-9da4-3a8845d0a8a9/graceful-termination/0.log"