Jul 24 17:17:56.836927 ip-10-0-137-105 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 24 17:17:56.836937 ip-10-0-137-105 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 24 17:17:56.836945 ip-10-0-137-105 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 24 17:17:56.837234 ip-10-0-137-105 systemd[1]: Failed to start Kubernetes Kubelet. Jul 24 17:18:07.070765 ip-10-0-137-105 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 24 17:18:07.070787 ip-10-0-137-105 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e70f23d61cae4d1eacb9e363de6878c8 -- Jul 24 17:20:02.201849 ip-10-0-137-105 systemd[1]: Starting Kubernetes Kubelet... Jul 24 17:20:02.661836 ip-10-0-137-105 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 17:20:02.661836 ip-10-0-137-105 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 17:20:02.661836 ip-10-0-137-105 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 17:20:02.661836 ip-10-0-137-105 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 17:20:02.661836 ip-10-0-137-105 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 17:20:02.664012 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.663919 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 17:20:02.669735 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669716 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:20:02.669735 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669735 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669739 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669743 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669747 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669750 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669754 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669757 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669759 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669762 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669765 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669767 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669770 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669772 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669775 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669777 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669779 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669782 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669786 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669789 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669792 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:20:02.669812 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669795 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669797 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669800 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669803 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669805 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669808 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669811 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669813 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669816 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669819 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669822 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669825 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669834 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669838 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669840 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669844 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669846 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669849 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669852 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:20:02.670396 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669855 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669859 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669862 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669865 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669867 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669870 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669872 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669875 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669877 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669879 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669882 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669884 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669886 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669889 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669892 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669894 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669897 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669900 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669902 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669905 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:20:02.671203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669908 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669910 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669913 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669915 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669918 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669920 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669923 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669926 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669928 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669931 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669933 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669938 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669951 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669956 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669959 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669962 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669965 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669968 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669971 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:20:02.671717 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669973 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669976 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669978 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669980 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669983 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669985 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669989 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669991 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669996 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.669999 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.670001 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.670004 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.670007 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.670009 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.670012 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.670014 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.670017 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.670019 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.670022 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:20:02.672181 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.670024 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672588 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672598 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672601 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672604 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672607 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672610 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672615 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672619 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672622 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672625 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672627 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672630 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672632 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672635 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672637 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672639 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672642 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672644 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:20:02.672651 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672648 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672650 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672653 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672656 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672659 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672662 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672664 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672667 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672669 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672671 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672674 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672676 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672678 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672681 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672683 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672686 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672688 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672691 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672693 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672696 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672699 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:20:02.673121 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672701 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672704 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672706 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672708 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672711 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672713 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672717 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672720 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672723 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672725 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672728 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672731 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672734 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672737 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672740 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672742 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672745 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672747 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672750 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:20:02.673635 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672755 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672758 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672761 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672764 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672767 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672769 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672772 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672774 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672777 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672780 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672782 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672785 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672788 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672790 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672793 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672796 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672798 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672800 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672803 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672805 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:20:02.674108 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672807 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672809 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672812 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672814 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672817 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672820 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672823 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672826 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672828 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672831 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672833 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672836 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672838 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672840 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672843 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672845 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672848 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672850 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672852 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672854 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:20:02.674626 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.672857 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673240 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673244 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673247 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673250 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673252 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673255 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673257 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673260 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673262 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673265 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673268 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673271 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673274 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673276 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673278 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673281 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673284 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673288 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:20:02.675104 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673291 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673294 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673296 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673298 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673301 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673304 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673306 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673309 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673311 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673314 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673316 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673318 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673321 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673323 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673325 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673328 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673331 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673334 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673336 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673338 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:20:02.675583 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673341 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673343 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673346 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673348 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673351 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673353 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673356 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673358 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673361 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673363 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673365 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673369 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673371 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673373 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673376 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673378 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673381 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673383 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673386 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673388 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:20:02.676107 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673391 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673393 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673395 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673398 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673400 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673402 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673405 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673407 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673409 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673413 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673416 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673419 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673422 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673424 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673427 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673429 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673434 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673436 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673439 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:20:02.676610 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673443 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673445 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673448 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673451 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673454 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673458 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673462 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673464 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673467 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673470 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673472 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673475 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673477 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673479 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673500 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673504 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673507 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673525 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673528 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:20:02.677051 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673531 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673534 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.673536 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673606 2559 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673614 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673628 2559 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673632 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673636 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673639 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673643 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673649 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673652 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673655 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673659 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673662 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673665 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673668 2559 flags.go:64] FLAG: --cgroup-root="" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673671 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673675 2559 flags.go:64] FLAG: --client-ca-file="" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673678 2559 flags.go:64] FLAG: --cloud-provider="external" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673681 2559 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673686 2559 flags.go:64] FLAG: --cluster-domain="" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673688 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673691 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 17:20:02.677573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673694 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673697 2559 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673705 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673708 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673711 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673714 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673717 2559 flags.go:64] FLAG: --contention-profiling="false" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673720 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673723 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673726 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673729 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673733 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673736 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673739 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673742 2559 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673745 2559 flags.go:64] FLAG: --enable-server="true" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673747 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673752 2559 flags.go:64] FLAG: --event-burst="100" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673756 2559 flags.go:64] FLAG: --event-qps="50" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673758 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673761 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673764 2559 flags.go:64] FLAG: --eviction-hard="" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673768 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673771 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673774 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 17:20:02.678175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673777 2559 flags.go:64] FLAG: --eviction-soft="" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673780 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673783 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673787 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673789 2559 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673792 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673795 2559 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673797 2559 flags.go:64] FLAG: --feature-gates="" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673801 2559 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673804 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673806 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673809 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673812 2559 flags.go:64] FLAG: --healthz-port="10248" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673815 2559 flags.go:64] FLAG: --help="false" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673818 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-137-105.ec2.internal" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673821 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673823 2559 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673826 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673829 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673832 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673835 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673838 2559 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673841 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 17:20:02.678797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673843 2559 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673847 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673852 2559 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673854 2559 flags.go:64] FLAG: --kube-reserved="" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673857 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673860 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673863 2559 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673866 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673868 2559 flags.go:64] FLAG: --lock-file="" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673871 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673874 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673876 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673882 2559 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673885 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673888 2559 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673890 2559 flags.go:64] FLAG: --logging-format="text" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673893 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673896 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673899 2559 flags.go:64] FLAG: --manifest-url="" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673901 2559 flags.go:64] FLAG: --manifest-url-header="" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673905 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673908 2559 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673912 2559 flags.go:64] FLAG: --max-pods="110" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673915 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673918 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 17:20:02.679372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673920 2559 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673923 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673926 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673929 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673932 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673939 2559 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673950 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673954 2559 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673959 2559 flags.go:64] FLAG: --pod-cidr="" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673963 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673969 2559 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673972 2559 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673974 2559 flags.go:64] FLAG: --pods-per-core="0" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673977 2559 flags.go:64] FLAG: --port="10250" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673980 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673983 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d6fcf57f1afbb657" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673986 2559 flags.go:64] FLAG: --qos-reserved="" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673989 2559 flags.go:64] FLAG: --read-only-port="10255" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673992 2559 flags.go:64] FLAG: --register-node="true" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673995 2559 flags.go:64] FLAG: --register-with-taints="" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.673998 2559 flags.go:64] FLAG: --registry-burst="10" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674002 2559 flags.go:64] FLAG: --registry-qps="5" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674005 2559 flags.go:64] FLAG: --reserved-cpus="" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674008 2559 flags.go:64] FLAG: --reserved-memory="" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674011 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 17:20:02.680004 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674014 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674017 2559 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674019 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674022 2559 flags.go:64] FLAG: --runonce="false" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674025 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674028 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674030 2559 flags.go:64] FLAG: --seccomp-default="false" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674033 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674036 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674039 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674041 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674044 2559 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674047 2559 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674050 2559 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674052 2559 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674056 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674059 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674064 2559 flags.go:64] FLAG: --system-cgroups="" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674067 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674072 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674074 2559 flags.go:64] FLAG: --tls-cert-file="" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674077 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674081 2559 flags.go:64] FLAG: --tls-min-version="" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674084 2559 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674087 2559 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 17:20:02.680642 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674089 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674092 2559 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674095 2559 flags.go:64] FLAG: --v="2" Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674098 2559 flags.go:64] FLAG: --version="false" Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674103 2559 flags.go:64] FLAG: --vmodule="" Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674107 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.674110 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674228 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674231 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674234 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674237 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674240 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674243 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674246 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674248 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674251 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674254 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674256 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674259 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674261 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674263 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:20:02.681294 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674266 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674270 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674272 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674276 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674278 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674281 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674283 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674286 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674288 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674291 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674293 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674296 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674298 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674301 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674303 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674305 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674315 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674318 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674322 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674326 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:20:02.681839 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674328 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674331 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674333 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674336 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674339 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674341 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674344 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674346 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674349 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674352 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674354 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674357 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674359 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674363 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674366 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674370 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674372 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674374 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674377 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674379 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:20:02.682332 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674382 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674385 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674389 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674392 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674395 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674398 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674401 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674403 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674406 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674411 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674414 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674416 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674419 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674421 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674423 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674426 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674428 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674432 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674435 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:20:02.682844 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674437 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674440 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674442 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674445 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674448 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674450 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674454 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674456 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674460 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674462 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674465 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674467 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674469 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674472 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674474 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674477 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674480 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674482 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674485 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674487 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:20:02.683307 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674490 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674492 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674495 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674498 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674501 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.674503 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.675093 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.682263 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.682282 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.682302 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.682307 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682365 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682370 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682374 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682377 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:20:02.683820 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682381 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682384 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682386 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682389 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682392 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682394 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682397 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682400 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682402 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682404 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682407 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682409 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682411 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682414 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682416 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682419 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682421 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682424 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682426 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682429 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:20:02.684205 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682433 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682435 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682439 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682443 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682446 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682449 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682452 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682455 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682458 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682460 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682463 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682466 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682469 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682471 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682474 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682477 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682479 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682482 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682484 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:20:02.684764 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682487 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682490 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682492 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682495 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682497 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682500 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682502 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682505 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682507 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682524 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682527 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682529 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682532 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682535 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682538 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682541 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682544 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682546 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682550 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682553 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:20:02.685229 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682556 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682558 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682561 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682564 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682567 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682569 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682573 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682576 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682579 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682583 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682586 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682589 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682591 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682594 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682597 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682600 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682603 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682605 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682608 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:20:02.685730 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682611 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682613 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682615 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682618 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682622 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682625 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682628 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682631 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682635 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682637 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682640 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682643 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682645 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682648 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682650 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682653 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:20:02.686203 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682655 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.682660 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682776 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682780 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682783 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682786 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682789 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682792 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682794 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682797 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682799 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682802 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682804 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682807 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682810 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:20:02.686607 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682812 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682815 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682817 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682819 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682822 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682824 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682826 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682829 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682832 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682835 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682838 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682840 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682843 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682845 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682848 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682851 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682854 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682857 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682860 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682862 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:20:02.686983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682865 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682867 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682870 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682872 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682875 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682877 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682879 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682882 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682884 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682887 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682889 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682892 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682894 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682897 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682899 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682902 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682904 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682906 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682909 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682913 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:20:02.687530 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682916 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682918 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682920 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682924 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682927 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682931 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682934 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682936 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682939 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682941 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682944 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682946 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682949 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682951 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682954 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682956 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682959 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682961 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682965 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:20:02.688019 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682968 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682971 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682974 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682976 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682979 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682982 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682984 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682987 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682989 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682991 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682994 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.682997 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683000 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683003 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683006 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683009 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683011 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683014 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683017 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683020 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:20:02.688473 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683022 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:20:02.688977 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683026 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:20:02.688977 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683029 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:20:02.688977 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683032 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:20:02.688977 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683035 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:20:02.688977 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683038 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:20:02.688977 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:02.683041 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:20:02.688977 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.683046 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 17:20:02.688977 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.683174 2559 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 17:20:02.688977 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.685369 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 17:20:02.688977 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.686664 2559 server.go:1019] "Starting client certificate rotation" Jul 24 17:20:02.688977 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.686765 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 17:20:02.688977 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.686804 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 17:20:02.689795 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.689778 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 17:20:02.690949 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.690930 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 17:20:02.702347 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.702322 2559 log.go:25] "Validated CRI v1 runtime API" Jul 24 17:20:02.706525 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.706493 2559 log.go:25] "Validated CRI v1 image API" Jul 24 17:20:02.707293 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.707268 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 17:20:02.710459 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.710437 2559 fs.go:135] Filesystem UUIDs: map[505ce524-c094-4e82-bfef-e62c0db87c47:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a8c2fe55-f6fc-40cf-bb72-396f7fb4bf1d:/dev/nvme0n1p4] Jul 24 17:20:02.710539 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.710460 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 17:20:02.716661 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.716535 2559 manager.go:216] Machine: {Timestamp:2026-07-24 17:20:02.714990988 +0000 UTC m=+0.366882452 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090964 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ac8771743852fbf9c1f6a828d4081 SystemUUID:ec2ac877-1743-852f-bf9c-1f6a828d4081 BootID:e70f23d6-1cae-4d1e-acb9-e363de6878c8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d7:86:6f:21:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d7:86:6f:21:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:87:15:c8:d4:57 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 17:20:02.716661 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.716648 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 17:20:02.716785 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.716741 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 17:20:02.720778 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.720740 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 17:20:02.720930 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.720781 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 17:20:02.721493 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.721483 2559 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 17:20:02.721568 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.721495 2559 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 17:20:02.721568 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.721502 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 17:20:02.722297 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.722276 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 17:20:02.722597 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.722584 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 17:20:02.723448 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.723431 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 17:20:02.725386 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.725367 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 24 17:20:02.725505 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.725495 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 17:20:02.725560 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.725538 2559 kubelet.go:486] "Attempting to sync node with API server" Jul 24 17:20:02.725560 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.725547 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 17:20:02.725633 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.725563 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 17:20:02.725633 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.725572 2559 kubelet.go:398] "Adding apiserver pod source" Jul 24 17:20:02.725633 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.725581 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 17:20:02.725996 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.725982 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 17:20:02.726037 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.726006 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 17:20:02.729058 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.729037 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 17:20:02.730967 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.730953 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 17:20:02.731027 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.730983 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 17:20:02.732586 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.732569 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 17:20:02.732586 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.732589 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 17:20:02.732719 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.732597 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 17:20:02.732719 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.732603 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 17:20:02.732719 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.732612 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 17:20:02.732719 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.732617 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 17:20:02.732719 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.732625 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 17:20:02.732719 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.732631 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 17:20:02.732719 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.732638 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 17:20:02.732719 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.732644 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 17:20:02.732719 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.732652 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 17:20:02.732719 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.732659 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 17:20:02.734165 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.734147 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 17:20:02.734165 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.734165 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 17:20:02.735909 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:02.735877 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 17:20:02.735909 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:02.735884 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 17:20:02.737990 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.737965 2559 server.go:1268] "Started kubelet" Jul 24 17:20:02.739003 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.738965 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 17:20:02.739149 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.739104 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 17:20:02.739144 ip-10-0-137-105 systemd[1]: Started Kubernetes Kubelet. Jul 24 17:20:02.739290 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.739183 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 17:20:02.741105 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.741088 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 17:20:02.742085 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.742059 2559 server.go:310] "Adding debug handlers to kubelet server" Jul 24 17:20:02.745811 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:02.745787 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 17:20:02.746322 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.746305 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xt7tb" Jul 24 17:20:02.747730 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.747711 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-105.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 17:20:02.748847 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:02.747703 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-105.ec2.internal.18c54976fdf4c612 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-105.ec2.internal,UID:ip-10-0-137-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-105.ec2.internal,},FirstTimestamp:2026-07-24 17:20:02.73791541 +0000 UTC m=+0.389806877,LastTimestamp:2026-07-24 17:20:02.73791541 +0000 UTC m=+0.389806877,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-105.ec2.internal,}" Jul 24 17:20:02.749654 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.749638 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 17:20:02.749767 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.749634 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 17:20:02.749767 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:02.749732 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-105.ec2.internal\" not found" Jul 24 17:20:02.750327 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.750230 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 17:20:02.750327 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.750241 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 17:20:02.750463 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.750345 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 17:20:02.750463 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.750423 2559 reconstruct.go:97] "Volume reconstruction finished" Jul 24 17:20:02.750463 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.750434 2559 reconciler.go:29] "Reconciler: start to sync state" Jul 24 17:20:02.750793 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.750773 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 17:20:02.750793 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.750792 2559 factory.go:55] Registering systemd factory Jul 24 17:20:02.750931 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.750799 2559 factory.go:223] Registration of the systemd container factory successfully Jul 24 17:20:02.751013 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.750997 2559 factory.go:153] Registering CRI-O factory Jul 24 17:20:02.751088 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.751018 2559 factory.go:223] Registration of the crio container factory successfully Jul 24 17:20:02.751088 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.751043 2559 factory.go:103] Registering Raw factory Jul 24 17:20:02.751088 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.751060 2559 manager.go:1196] Started watching for new ooms in manager Jul 24 17:20:02.751531 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.751498 2559 manager.go:349] Starting recovery of all containers Jul 24 17:20:02.754251 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.754226 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xt7tb" Jul 24 17:20:02.756080 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:02.756059 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 17:20:02.756183 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:02.756138 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 17:20:02.761827 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.761702 2559 manager.go:354] Recovery completed Jul 24 17:20:02.766399 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.766385 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:20:02.769198 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.769175 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:20:02.769287 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.769214 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:20:02.769287 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.769228 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:20:02.769880 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.769867 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 17:20:02.769880 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.769875 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:20:02.769880 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.769879 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 17:20:02.770002 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.769894 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 24 17:20:02.770581 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.770567 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:20:02.770630 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.770592 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:20:02.770630 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.770601 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:20:02.771898 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.771888 2559 policy_none.go:49] "None policy: Start" Jul 24 17:20:02.771932 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.771903 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 17:20:02.771932 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.771912 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 17:20:02.773706 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.773693 2559 policy_none.go:47] "Start" Jul 24 17:20:02.810336 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.810318 2559 manager.go:337] "Starting Device Plugin manager" Jul 24 17:20:02.810466 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:02.810352 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 17:20:02.810466 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.810362 2559 server.go:87] "Starting device plugin registration server" Jul 24 17:20:02.810696 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.810684 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 17:20:02.810768 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.810700 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 17:20:02.810819 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.810806 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 17:20:02.810927 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.810910 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 17:20:02.810927 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.810926 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 17:20:02.811460 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:02.811436 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 17:20:02.811577 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:02.811497 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-105.ec2.internal\" not found" Jul 24 17:20:02.911799 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.911758 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:20:02.913011 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.912804 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:20:02.913011 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.912833 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:20:02.913011 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.912843 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:20:02.913011 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.912877 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-105.ec2.internal" Jul 24 17:20:02.919560 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.919542 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-105.ec2.internal" Jul 24 17:20:02.919625 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:02.919568 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-105.ec2.internal\": node \"ip-10-0-137-105.ec2.internal\" not found" Jul 24 17:20:02.935262 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.935230 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 17:20:02.936671 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.936650 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 17:20:02.936776 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.936693 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 17:20:02.936776 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.936715 2559 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 17:20:02.936776 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:02.936747 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 17:20:02.940730 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:02.940707 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:20:02.978450 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:02.978421 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-105.ec2.internal\" not found" Jul 24 17:20:03.036885 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.036855 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-105.ec2.internal"] Jul 24 17:20:03.036980 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.036951 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:20:03.037950 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.037927 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:20:03.038015 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.037971 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:20:03.038015 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.037982 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:20:03.039280 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.039266 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:20:03.039397 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.039381 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.039451 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.039417 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:20:03.039991 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.039975 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:20:03.040064 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.040007 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:20:03.040064 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.040017 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:20:03.040064 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.039975 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:20:03.040145 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.040086 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:20:03.040145 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.040097 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:20:03.041251 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.041233 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.041340 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.041258 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:20:03.041866 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.041852 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:20:03.041931 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.041877 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:20:03.041931 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.041889 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:20:03.052530 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.052495 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca99be3d06dd468dc82690167e3e51d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal\" (UID: \"ca99be3d06dd468dc82690167e3e51d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.052608 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.052533 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca99be3d06dd468dc82690167e3e51d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal\" (UID: \"ca99be3d06dd468dc82690167e3e51d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.052608 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.052575 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4b95910892a229c48b40c8bd7c3819e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-105.ec2.internal\" (UID: \"a4b95910892a229c48b40c8bd7c3819e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.071438 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:03.071406 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-105.ec2.internal\" not found" node="ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.076082 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:03.076063 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-105.ec2.internal\" not found" node="ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.079038 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:03.079022 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-105.ec2.internal\" not found" Jul 24 17:20:03.153279 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.153243 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca99be3d06dd468dc82690167e3e51d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal\" (UID: \"ca99be3d06dd468dc82690167e3e51d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.153279 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.153278 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca99be3d06dd468dc82690167e3e51d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal\" (UID: \"ca99be3d06dd468dc82690167e3e51d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.153503 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.153304 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4b95910892a229c48b40c8bd7c3819e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-105.ec2.internal\" (UID: \"a4b95910892a229c48b40c8bd7c3819e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.153503 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.153353 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca99be3d06dd468dc82690167e3e51d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal\" (UID: \"ca99be3d06dd468dc82690167e3e51d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.153503 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.153371 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca99be3d06dd468dc82690167e3e51d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal\" (UID: \"ca99be3d06dd468dc82690167e3e51d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.153503 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.153353 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4b95910892a229c48b40c8bd7c3819e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-105.ec2.internal\" (UID: \"a4b95910892a229c48b40c8bd7c3819e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.179580 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:03.179478 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-105.ec2.internal\" not found" Jul 24 17:20:03.280416 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:03.280381 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-105.ec2.internal\" not found" Jul 24 17:20:03.373925 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.373892 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.379855 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.379837 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.380904 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:03.380888 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-105.ec2.internal\" not found" Jul 24 17:20:03.481048 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:03.480952 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-105.ec2.internal\" not found" Jul 24 17:20:03.581346 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:03.581304 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-105.ec2.internal\" not found" Jul 24 17:20:03.683102 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:03.683069 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-105.ec2.internal\" not found" Jul 24 17:20:03.686449 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.686423 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 17:20:03.686643 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.686625 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 17:20:03.759676 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.759573 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 17:15:03 +0000 UTC" deadline="2028-03-22 23:17:50.05781847 +0000 UTC" Jul 24 17:20:03.759676 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.759610 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14573h57m46.29821394s" Jul 24 17:20:03.783655 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:03.783613 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-105.ec2.internal\" not found" Jul 24 17:20:03.875937 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.875903 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:20:03.907500 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.907473 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:20:03.942186 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:03.942148 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b95910892a229c48b40c8bd7c3819e.slice/crio-b013fa2d1c74519b0a2a08f74beeb8069c8617a079b262c77238d8520125549d WatchSource:0}: Error finding container b013fa2d1c74519b0a2a08f74beeb8069c8617a079b262c77238d8520125549d: Status 404 returned error can't find the container with id b013fa2d1c74519b0a2a08f74beeb8069c8617a079b262c77238d8520125549d Jul 24 17:20:03.942833 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:03.942798 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca99be3d06dd468dc82690167e3e51d9.slice/crio-b8f4fce193aa9025d9a5df237885bc73d1791723f39478fe8c299b08211075ac WatchSource:0}: Error finding container b8f4fce193aa9025d9a5df237885bc73d1791723f39478fe8c299b08211075ac: Status 404 returned error can't find the container with id b8f4fce193aa9025d9a5df237885bc73d1791723f39478fe8c299b08211075ac Jul 24 17:20:03.947931 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.947914 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:20:03.950729 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.950706 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.962977 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.962953 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 17:20:03.965277 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.965259 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-105.ec2.internal" Jul 24 17:20:03.973219 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:03.973201 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 17:20:04.129867 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.129774 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:20:04.728029 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.727751 2559 apiserver.go:52] "Watching apiserver" Jul 24 17:20:04.736237 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.735912 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 17:20:04.737103 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.737077 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-bzsdh","openshift-ovn-kubernetes/ovnkube-node-pz4sn","kube-system/global-pull-secret-syncer-7bpxv","kube-system/kube-apiserver-proxy-ip-10-0-137-105.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal","openshift-multus/multus-additional-cni-plugins-r66jf","openshift-network-diagnostics/network-check-target-k42w8","openshift-network-operator/iptables-alerter-44xrk","kube-system/konnectivity-agent-d9bmx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq","openshift-cluster-node-tuning-operator/tuned-cwrz4","openshift-dns/node-resolver-ds7bq","openshift-image-registry/node-ca-499zx","openshift-multus/multus-fw5j2"] Jul 24 17:20:04.739057 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.739036 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.741099 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.741074 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.741812 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.741791 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 17:20:04.741899 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.741795 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l5bmf\"" Jul 24 17:20:04.742281 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.742260 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 17:20:04.742506 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.742488 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 17:20:04.743146 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.743129 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:04.743213 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:04.743195 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:04.743698 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.743678 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:20:04.744302 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.744288 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 17:20:04.744543 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.744518 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dm8xd\"" Jul 24 17:20:04.745325 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.745305 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.747619 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.747581 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-44xrk" Jul 24 17:20:04.749478 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.749456 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 17:20:04.749823 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.749611 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sw4nv\"" Jul 24 17:20:04.749823 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.749615 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 17:20:04.749823 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.749629 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 17:20:04.749823 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.749739 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 17:20:04.750536 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.750505 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 17:20:04.750802 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.750767 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 17:20:04.757939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.751202 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.757939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.752259 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d9bmx" Jul 24 17:20:04.757939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.753113 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xjbm4\"" Jul 24 17:20:04.757939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.753538 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 17:20:04.757939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.754193 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:20:04.757939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.755768 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 17:20:04.757939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.756460 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gq2xk\"" Jul 24 17:20:04.757939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.756807 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j4lf8\"" Jul 24 17:20:04.757939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.757482 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 17:20:04.757939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.757503 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-499zx" Jul 24 17:20:04.757939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.757505 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 17:20:04.757939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.757528 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 17:20:04.759395 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.759374 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.760709 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.760514 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p4qkw\"" Jul 24 17:20:04.760709 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.760545 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 17:20:04.760868 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.760710 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 17:20:04.761128 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.761105 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 17:20:04.762113 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762011 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-sys\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.762113 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762045 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64ee13b6-1701-4a39-8160-1a98f36410c3-tmp\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.762113 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762082 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe98c39f-4996-4c21-977f-2545fef7d6b7-multus-daemon-config\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.762336 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762132 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/812e91be-8ccb-4e8d-a3d8-185280698b0b-cni-binary-copy\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.762336 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762198 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-etc-selinux\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.762336 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762252 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-systemd\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.762336 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762278 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/812e91be-8ccb-4e8d-a3d8-185280698b0b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.762336 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762328 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-system-cni-dir\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.762624 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762353 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-run-multus-certs\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.762624 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762377 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.762624 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762420 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-device-dir\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.762624 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762447 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-sys-fs\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.762624 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762472 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-kubernetes\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.762624 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762496 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-run\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.762624 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762522 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f6e8751-f60c-4f88-a369-9158d269f5ff-kubelet-config\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:04.762624 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762546 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f6e8751-f60c-4f88-a369-9158d269f5ff-dbus\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:04.762624 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762612 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762630 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e621415d-ea2d-4520-8353-752e48ab5b39-konnectivity-ca\") pod \"konnectivity-agent-d9bmx\" (UID: \"e621415d-ea2d-4520-8353-752e48ab5b39\") " pod="kube-system/konnectivity-agent-d9bmx" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762660 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-multus-socket-dir-parent\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762687 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-var-lib-cni-bin\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762713 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-hostroot\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762740 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcnr8\" (UniqueName: \"kubernetes.io/projected/64ee13b6-1701-4a39-8160-1a98f36410c3-kube-api-access-pcnr8\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762770 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-multus-cni-dir\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762798 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-var-lib-kubelet\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762826 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762874 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-sysconfig\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762899 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-run-netns\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762922 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e-host-slash\") pod \"iptables-alerter-44xrk\" (UID: \"ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e\") " pod="openshift-network-operator/iptables-alerter-44xrk" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.762979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-modprobe-d\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763005 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/812e91be-8ccb-4e8d-a3d8-185280698b0b-os-release\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763039 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xljz\" (UniqueName: \"kubernetes.io/projected/812e91be-8ccb-4e8d-a3d8-185280698b0b-kube-api-access-2xljz\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.763082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763066 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-registration-dir\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763099 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-sysctl-d\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763128 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-var-lib-kubelet\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763165 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-os-release\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763191 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-sysctl-conf\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763220 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-var-lib-cni-multus\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763247 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-etc-kubernetes\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763275 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzzmm\" (UniqueName: \"kubernetes.io/projected/ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e-kube-api-access-mzzmm\") pod \"iptables-alerter-44xrk\" (UID: \"ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e\") " pod="openshift-network-operator/iptables-alerter-44xrk" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763307 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-lib-modules\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763449 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e-iptables-alerter-script\") pod \"iptables-alerter-44xrk\" (UID: \"ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e\") " pod="openshift-network-operator/iptables-alerter-44xrk" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763473 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/812e91be-8ccb-4e8d-a3d8-185280698b0b-cnibin\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763500 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpk8c\" (UniqueName: \"kubernetes.io/projected/067c1ab7-d33d-4d97-a434-4bcb7de77abb-kube-api-access-lpk8c\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763520 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-host\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763536 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28qh9\" (UniqueName: \"kubernetes.io/projected/fe98c39f-4996-4c21-977f-2545fef7d6b7-kube-api-access-28qh9\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763551 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/812e91be-8ccb-4e8d-a3d8-185280698b0b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763577 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-socket-dir\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763625 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-multus-conf-dir\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.763928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763636 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 17:20:04.764616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763652 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-tuned\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.764616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763697 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-cnibin\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.764616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763721 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe98c39f-4996-4c21-977f-2545fef7d6b7-cni-binary-copy\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.764616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763747 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-run-k8s-cni-cncf-io\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.764616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763776 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e621415d-ea2d-4520-8353-752e48ab5b39-agent-certs\") pod \"konnectivity-agent-d9bmx\" (UID: \"e621415d-ea2d-4520-8353-752e48ab5b39\") " pod="kube-system/konnectivity-agent-d9bmx" Jul 24 17:20:04.764616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763804 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/812e91be-8ccb-4e8d-a3d8-185280698b0b-system-cni-dir\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.764616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.763832 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/812e91be-8ccb-4e8d-a3d8-185280698b0b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.764616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.764032 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 17:20:04.764616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.764238 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 17:20:04.764616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.764247 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 17:20:04.764616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.764262 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 17:20:04.764616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.764397 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t5mc4\"" Jul 24 17:20:04.764616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.764601 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:04.765309 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:04.764699 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:04.765661 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.765533 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:04.765661 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.765602 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ds7bq" Jul 24 17:20:04.765661 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:04.765621 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:04.768712 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.768682 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d8hhn\"" Jul 24 17:20:04.768848 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.768833 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 17:20:04.768987 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.768973 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 17:20:04.774267 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.774243 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 17:20:04.799572 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.799069 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bfm87" Jul 24 17:20:04.808963 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.808932 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bfm87" Jul 24 17:20:04.853496 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.853460 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 17:20:04.864157 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864121 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-multus-cni-dir\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.864339 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-var-lib-kubelet\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.864339 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864193 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:04.864339 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864241 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-sysconfig\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.864339 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864277 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-run-netns\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.864339 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864304 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e-host-slash\") pod \"iptables-alerter-44xrk\" (UID: \"ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e\") " pod="openshift-network-operator/iptables-alerter-44xrk" Jul 24 17:20:04.864339 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864326 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96-hosts-file\") pod \"node-resolver-ds7bq\" (UID: \"dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96\") " pod="openshift-dns/node-resolver-ds7bq" Jul 24 17:20:04.864680 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864343 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22pzx\" (UniqueName: \"kubernetes.io/projected/7a498a6c-63f1-47ec-a1b5-f04d1b2230ff-kube-api-access-22pzx\") pod \"node-ca-499zx\" (UID: \"7a498a6c-63f1-47ec-a1b5-f04d1b2230ff\") " pod="openshift-image-registry/node-ca-499zx" Jul 24 17:20:04.864680 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864377 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-etc-openvswitch\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.864680 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864393 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-run-openvswitch\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.864680 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-multus-cni-dir\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.864680 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864466 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-sysconfig\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.864680 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864500 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-run-netns\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.864680 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864524 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e-host-slash\") pod \"iptables-alerter-44xrk\" (UID: \"ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e\") " pod="openshift-network-operator/iptables-alerter-44xrk" Jul 24 17:20:04.864680 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864578 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-var-lib-kubelet\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.864680 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:04.864533 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:20:04.864680 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:04.864668 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret podName:5f6e8751-f60c-4f88-a369-9158d269f5ff nodeName:}" failed. No retries permitted until 2026-07-24 17:20:05.364650255 +0000 UTC m=+3.016541723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret") pod "global-pull-secret-syncer-7bpxv" (UID: "5f6e8751-f60c-4f88-a369-9158d269f5ff") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864410 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-modprobe-d\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864804 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-modprobe-d\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864827 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/812e91be-8ccb-4e8d-a3d8-185280698b0b-os-release\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864848 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xljz\" (UniqueName: \"kubernetes.io/projected/812e91be-8ccb-4e8d-a3d8-185280698b0b-kube-api-access-2xljz\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864876 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a498a6c-63f1-47ec-a1b5-f04d1b2230ff-host\") pod \"node-ca-499zx\" (UID: \"7a498a6c-63f1-47ec-a1b5-f04d1b2230ff\") " pod="openshift-image-registry/node-ca-499zx" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864894 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-run-ovn\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.864971 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/812e91be-8ccb-4e8d-a3d8-185280698b0b-os-release\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865009 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-cni-bin\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865046 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-registration-dir\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865070 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-sysctl-d\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865092 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-var-lib-kubelet\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865123 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-os-release\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865139 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96-tmp-dir\") pod \"node-resolver-ds7bq\" (UID: \"dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96\") " pod="openshift-dns/node-resolver-ds7bq" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865154 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-sysctl-conf\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865166 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-sysctl-d\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.865187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865187 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-var-lib-cni-multus\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865212 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-etc-kubernetes\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865220 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-var-lib-kubelet\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865231 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzzmm\" (UniqueName: \"kubernetes.io/projected/ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e-kube-api-access-mzzmm\") pod \"iptables-alerter-44xrk\" (UID: \"ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e\") " pod="openshift-network-operator/iptables-alerter-44xrk" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865225 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-os-release\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865248 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-run-netns\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865276 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-lib-modules\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865295 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e-iptables-alerter-script\") pod \"iptables-alerter-44xrk\" (UID: \"ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e\") " pod="openshift-network-operator/iptables-alerter-44xrk" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865311 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/812e91be-8ccb-4e8d-a3d8-185280698b0b-cnibin\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865313 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-sysctl-conf\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865343 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2wp9\" (UniqueName: \"kubernetes.io/projected/dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96-kube-api-access-n2wp9\") pod \"node-resolver-ds7bq\" (UID: \"dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96\") " pod="openshift-dns/node-resolver-ds7bq" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865298 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-etc-kubernetes\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865365 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31e846fc-447f-4c0a-b40e-a18bb97e7841-ovn-node-metrics-cert\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865384 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpk8c\" (UniqueName: \"kubernetes.io/projected/067c1ab7-d33d-4d97-a434-4bcb7de77abb-kube-api-access-lpk8c\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865355 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/812e91be-8ccb-4e8d-a3d8-185280698b0b-cnibin\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865401 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-host\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865160 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-registration-dir\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.865998 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865426 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-run-systemd\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865438 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-lib-modules\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865445 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-var-lib-openvswitch\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865444 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-var-lib-cni-multus\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865461 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31e846fc-447f-4c0a-b40e-a18bb97e7841-env-overrides\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865478 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28qh9\" (UniqueName: \"kubernetes.io/projected/fe98c39f-4996-4c21-977f-2545fef7d6b7-kube-api-access-28qh9\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865508 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/812e91be-8ccb-4e8d-a3d8-185280698b0b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865526 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-socket-dir\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865565 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-multus-conf-dir\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865589 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-tuned\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865613 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-cnibin\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865618 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-host\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865637 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe98c39f-4996-4c21-977f-2545fef7d6b7-cni-binary-copy\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-run-k8s-cni-cncf-io\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865615 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-multus-conf-dir\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865683 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e621415d-ea2d-4520-8353-752e48ab5b39-agent-certs\") pod \"konnectivity-agent-d9bmx\" (UID: \"e621415d-ea2d-4520-8353-752e48ab5b39\") " pod="kube-system/konnectivity-agent-d9bmx" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865702 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/812e91be-8ccb-4e8d-a3d8-185280698b0b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.866960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865709 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-socket-dir\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865731 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/812e91be-8ccb-4e8d-a3d8-185280698b0b-system-cni-dir\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/812e91be-8ccb-4e8d-a3d8-185280698b0b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865768 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-cnibin\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865778 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-run-k8s-cni-cncf-io\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865792 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-node-log\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865824 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/812e91be-8ccb-4e8d-a3d8-185280698b0b-system-cni-dir\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865884 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-sys\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865898 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865928 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64ee13b6-1701-4a39-8160-1a98f36410c3-tmp\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865957 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe98c39f-4996-4c21-977f-2545fef7d6b7-multus-daemon-config\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.865980 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/812e91be-8ccb-4e8d-a3d8-185280698b0b-cni-binary-copy\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866004 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7a498a6c-63f1-47ec-a1b5-f04d1b2230ff-serviceca\") pod \"node-ca-499zx\" (UID: \"7a498a6c-63f1-47ec-a1b5-f04d1b2230ff\") " pod="openshift-image-registry/node-ca-499zx" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866028 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-cni-netd\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866052 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866085 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31e846fc-447f-4c0a-b40e-a18bb97e7841-ovnkube-script-lib\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866111 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-etc-selinux\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.867705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866132 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-systemd\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866157 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s87tc\" (UniqueName: \"kubernetes.io/projected/505f4198-635a-4e18-847d-f85e84b9f70e-kube-api-access-s87tc\") pod \"network-metrics-daemon-bzsdh\" (UID: \"505f4198-635a-4e18-847d-f85e84b9f70e\") " pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866183 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpsbc\" (UniqueName: \"kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc\") pod \"network-check-target-k42w8\" (UID: \"352eb282-95b7-442d-9834-8703f5d0db9a\") " pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866188 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e-iptables-alerter-script\") pod \"iptables-alerter-44xrk\" (UID: \"ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e\") " pod="openshift-network-operator/iptables-alerter-44xrk" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866207 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-log-socket\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866228 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31e846fc-447f-4c0a-b40e-a18bb97e7841-ovnkube-config\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866258 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/812e91be-8ccb-4e8d-a3d8-185280698b0b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866287 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-system-cni-dir\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866285 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-sys\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866297 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe98c39f-4996-4c21-977f-2545fef7d6b7-cni-binary-copy\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866316 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-run-multus-certs\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866349 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt47p\" (UniqueName: \"kubernetes.io/projected/31e846fc-447f-4c0a-b40e-a18bb97e7841-kube-api-access-pt47p\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866386 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/812e91be-8ccb-4e8d-a3d8-185280698b0b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866407 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-system-cni-dir\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866410 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-etc-selinux\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866419 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-systemd\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866470 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.868435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866503 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-run-multus-certs\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866559 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-device-dir\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866598 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-device-dir\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866657 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-sys-fs\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866681 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866723 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-kubernetes\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866749 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-run\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866793 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-run\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866794 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-kubernetes\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866837 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/067c1ab7-d33d-4d97-a434-4bcb7de77abb-sys-fs\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866842 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-kubelet\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f6e8751-f60c-4f88-a369-9158d269f5ff-kubelet-config\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866931 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f6e8751-f60c-4f88-a369-9158d269f5ff-dbus\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866932 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/812e91be-8ccb-4e8d-a3d8-185280698b0b-cni-binary-copy\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866932 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe98c39f-4996-4c21-977f-2545fef7d6b7-multus-daemon-config\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866932 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/812e91be-8ccb-4e8d-a3d8-185280698b0b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866958 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e621415d-ea2d-4520-8353-752e48ab5b39-konnectivity-ca\") pod \"konnectivity-agent-d9bmx\" (UID: \"e621415d-ea2d-4520-8353-752e48ab5b39\") " pod="kube-system/konnectivity-agent-d9bmx" Jul 24 17:20:04.869083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866989 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs\") pod \"network-metrics-daemon-bzsdh\" (UID: \"505f4198-635a-4e18-847d-f85e84b9f70e\") " pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:04.869625 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.866997 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f6e8751-f60c-4f88-a369-9158d269f5ff-kubelet-config\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:04.869625 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.867022 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-run-ovn-kubernetes\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.869625 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.867035 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f6e8751-f60c-4f88-a369-9158d269f5ff-dbus\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:04.869625 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.867060 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-multus-socket-dir-parent\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.869625 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.867109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-var-lib-cni-bin\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.869625 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.867118 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-multus-socket-dir-parent\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.869625 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.867153 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-hostroot\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.869625 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.867187 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-host-var-lib-cni-bin\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.869625 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.867226 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-systemd-units\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.869625 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.867247 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe98c39f-4996-4c21-977f-2545fef7d6b7-hostroot\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.869625 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.867254 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-slash\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.869625 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.867285 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcnr8\" (UniqueName: \"kubernetes.io/projected/64ee13b6-1701-4a39-8160-1a98f36410c3-kube-api-access-pcnr8\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.869625 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.867510 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e621415d-ea2d-4520-8353-752e48ab5b39-konnectivity-ca\") pod \"konnectivity-agent-d9bmx\" (UID: \"e621415d-ea2d-4520-8353-752e48ab5b39\") " pod="kube-system/konnectivity-agent-d9bmx" Jul 24 17:20:04.870903 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.870878 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64ee13b6-1701-4a39-8160-1a98f36410c3-tmp\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.870903 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.870899 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64ee13b6-1701-4a39-8160-1a98f36410c3-etc-tuned\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.873541 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.873517 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e621415d-ea2d-4520-8353-752e48ab5b39-agent-certs\") pod \"konnectivity-agent-d9bmx\" (UID: \"e621415d-ea2d-4520-8353-752e48ab5b39\") " pod="kube-system/konnectivity-agent-d9bmx" Jul 24 17:20:04.876427 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.876405 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xljz\" (UniqueName: \"kubernetes.io/projected/812e91be-8ccb-4e8d-a3d8-185280698b0b-kube-api-access-2xljz\") pod \"multus-additional-cni-plugins-r66jf\" (UID: \"812e91be-8ccb-4e8d-a3d8-185280698b0b\") " pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:04.876908 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.876883 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzzmm\" (UniqueName: \"kubernetes.io/projected/ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e-kube-api-access-mzzmm\") pod \"iptables-alerter-44xrk\" (UID: \"ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e\") " pod="openshift-network-operator/iptables-alerter-44xrk" Jul 24 17:20:04.877861 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.877834 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28qh9\" (UniqueName: \"kubernetes.io/projected/fe98c39f-4996-4c21-977f-2545fef7d6b7-kube-api-access-28qh9\") pod \"multus-fw5j2\" (UID: \"fe98c39f-4996-4c21-977f-2545fef7d6b7\") " pod="openshift-multus/multus-fw5j2" Jul 24 17:20:04.878878 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.878780 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpk8c\" (UniqueName: \"kubernetes.io/projected/067c1ab7-d33d-4d97-a434-4bcb7de77abb-kube-api-access-lpk8c\") pod \"aws-ebs-csi-driver-node-96wlq\" (UID: \"067c1ab7-d33d-4d97-a434-4bcb7de77abb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:04.880139 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.880119 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcnr8\" (UniqueName: \"kubernetes.io/projected/64ee13b6-1701-4a39-8160-1a98f36410c3-kube-api-access-pcnr8\") pod \"tuned-cwrz4\" (UID: \"64ee13b6-1701-4a39-8160-1a98f36410c3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:04.941863 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.941799 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-105.ec2.internal" event={"ID":"a4b95910892a229c48b40c8bd7c3819e","Type":"ContainerStarted","Data":"b013fa2d1c74519b0a2a08f74beeb8069c8617a079b262c77238d8520125549d"} Jul 24 17:20:04.943259 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.943192 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal" event={"ID":"ca99be3d06dd468dc82690167e3e51d9","Type":"ContainerStarted","Data":"b8f4fce193aa9025d9a5df237885bc73d1791723f39478fe8c299b08211075ac"} Jul 24 17:20:04.967977 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.967932 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-run-openvswitch\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.967977 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.967981 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a498a6c-63f1-47ec-a1b5-f04d1b2230ff-host\") pod \"node-ca-499zx\" (UID: \"7a498a6c-63f1-47ec-a1b5-f04d1b2230ff\") " pod="openshift-image-registry/node-ca-499zx" Jul 24 17:20:04.968247 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968006 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-run-ovn\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968247 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968032 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-cni-bin\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968247 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968062 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96-tmp-dir\") pod \"node-resolver-ds7bq\" (UID: \"dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96\") " pod="openshift-dns/node-resolver-ds7bq" Jul 24 17:20:04.968247 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968075 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a498a6c-63f1-47ec-a1b5-f04d1b2230ff-host\") pod \"node-ca-499zx\" (UID: \"7a498a6c-63f1-47ec-a1b5-f04d1b2230ff\") " pod="openshift-image-registry/node-ca-499zx" Jul 24 17:20:04.968247 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968093 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-run-ovn\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968247 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-run-netns\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968247 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968139 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-run-netns\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968247 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968058 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-run-openvswitch\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968247 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968100 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-cni-bin\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968247 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968159 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2wp9\" (UniqueName: \"kubernetes.io/projected/dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96-kube-api-access-n2wp9\") pod \"node-resolver-ds7bq\" (UID: \"dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96\") " pod="openshift-dns/node-resolver-ds7bq" Jul 24 17:20:04.968247 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968202 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31e846fc-447f-4c0a-b40e-a18bb97e7841-ovn-node-metrics-cert\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968247 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968235 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-run-systemd\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968292 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-var-lib-openvswitch\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968320 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31e846fc-447f-4c0a-b40e-a18bb97e7841-env-overrides\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968358 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-node-log\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968419 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96-tmp-dir\") pod \"node-resolver-ds7bq\" (UID: \"dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96\") " pod="openshift-dns/node-resolver-ds7bq" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968395 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7a498a6c-63f1-47ec-a1b5-f04d1b2230ff-serviceca\") pod \"node-ca-499zx\" (UID: \"7a498a6c-63f1-47ec-a1b5-f04d1b2230ff\") " pod="openshift-image-registry/node-ca-499zx" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968437 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-var-lib-openvswitch\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968464 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-cni-netd\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968491 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-run-systemd\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968502 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968507 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-node-log\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968535 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31e846fc-447f-4c0a-b40e-a18bb97e7841-ovnkube-script-lib\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968566 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-cni-netd\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968636 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968693 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s87tc\" (UniqueName: \"kubernetes.io/projected/505f4198-635a-4e18-847d-f85e84b9f70e-kube-api-access-s87tc\") pod \"network-metrics-daemon-bzsdh\" (UID: \"505f4198-635a-4e18-847d-f85e84b9f70e\") " pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968739 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpsbc\" (UniqueName: \"kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc\") pod \"network-check-target-k42w8\" (UID: \"352eb282-95b7-442d-9834-8703f5d0db9a\") " pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968768 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-log-socket\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.968981 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968803 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31e846fc-447f-4c0a-b40e-a18bb97e7841-ovnkube-config\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968836 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt47p\" (UniqueName: \"kubernetes.io/projected/31e846fc-447f-4c0a-b40e-a18bb97e7841-kube-api-access-pt47p\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968869 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-kubelet\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs\") pod \"network-metrics-daemon-bzsdh\" (UID: \"505f4198-635a-4e18-847d-f85e84b9f70e\") " pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968928 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-run-ovn-kubernetes\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968962 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-systemd-units\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.968983 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31e846fc-447f-4c0a-b40e-a18bb97e7841-env-overrides\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.969028 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-slash\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.969080 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96-hosts-file\") pod \"node-resolver-ds7bq\" (UID: \"dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96\") " pod="openshift-dns/node-resolver-ds7bq" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.969110 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22pzx\" (UniqueName: \"kubernetes.io/projected/7a498a6c-63f1-47ec-a1b5-f04d1b2230ff-kube-api-access-22pzx\") pod \"node-ca-499zx\" (UID: \"7a498a6c-63f1-47ec-a1b5-f04d1b2230ff\") " pod="openshift-image-registry/node-ca-499zx" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.969137 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-etc-openvswitch\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.969132 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-run-ovn-kubernetes\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.969157 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31e846fc-447f-4c0a-b40e-a18bb97e7841-ovnkube-script-lib\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.969214 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-log-socket\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.969234 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-kubelet\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.969245 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96-hosts-file\") pod \"node-resolver-ds7bq\" (UID: \"dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96\") " pod="openshift-dns/node-resolver-ds7bq" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.969297 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-etc-openvswitch\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:04.969342 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:20:04.969973 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.969415 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-host-slash\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.970800 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:04.969476 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs podName:505f4198-635a-4e18-847d-f85e84b9f70e nodeName:}" failed. No retries permitted until 2026-07-24 17:20:05.469446384 +0000 UTC m=+3.121337837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs") pod "network-metrics-daemon-bzsdh" (UID: "505f4198-635a-4e18-847d-f85e84b9f70e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:20:04.970800 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.969592 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7a498a6c-63f1-47ec-a1b5-f04d1b2230ff-serviceca\") pod \"node-ca-499zx\" (UID: \"7a498a6c-63f1-47ec-a1b5-f04d1b2230ff\") " pod="openshift-image-registry/node-ca-499zx" Jul 24 17:20:04.970800 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.969610 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31e846fc-447f-4c0a-b40e-a18bb97e7841-systemd-units\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.970800 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.969963 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31e846fc-447f-4c0a-b40e-a18bb97e7841-ovnkube-config\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.971104 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.971086 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31e846fc-447f-4c0a-b40e-a18bb97e7841-ovn-node-metrics-cert\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.977535 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:04.977168 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:20:04.977535 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:04.977204 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:20:04.977535 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:04.977221 2559 projected.go:196] Error preparing data for projected volume kube-api-access-jpsbc for pod openshift-network-diagnostics/network-check-target-k42w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:20:04.977535 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:04.977294 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc podName:352eb282-95b7-442d-9834-8703f5d0db9a nodeName:}" failed. No retries permitted until 2026-07-24 17:20:05.477271782 +0000 UTC m=+3.129163263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jpsbc" (UniqueName: "kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc") pod "network-check-target-k42w8" (UID: "352eb282-95b7-442d-9834-8703f5d0db9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:20:04.978405 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.978352 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2wp9\" (UniqueName: \"kubernetes.io/projected/dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96-kube-api-access-n2wp9\") pod \"node-resolver-ds7bq\" (UID: \"dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96\") " pod="openshift-dns/node-resolver-ds7bq" Jul 24 17:20:04.978405 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.978392 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s87tc\" (UniqueName: \"kubernetes.io/projected/505f4198-635a-4e18-847d-f85e84b9f70e-kube-api-access-s87tc\") pod \"network-metrics-daemon-bzsdh\" (UID: \"505f4198-635a-4e18-847d-f85e84b9f70e\") " pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:04.979770 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.979747 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt47p\" (UniqueName: \"kubernetes.io/projected/31e846fc-447f-4c0a-b40e-a18bb97e7841-kube-api-access-pt47p\") pod \"ovnkube-node-pz4sn\" (UID: \"31e846fc-447f-4c0a-b40e-a18bb97e7841\") " pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:04.980503 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:04.980479 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22pzx\" (UniqueName: \"kubernetes.io/projected/7a498a6c-63f1-47ec-a1b5-f04d1b2230ff-kube-api-access-22pzx\") pod \"node-ca-499zx\" (UID: \"7a498a6c-63f1-47ec-a1b5-f04d1b2230ff\") " pod="openshift-image-registry/node-ca-499zx" Jul 24 17:20:05.055687 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.055640 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" Jul 24 17:20:05.063430 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.063405 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:20:05.068730 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.068700 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" Jul 24 17:20:05.077824 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.077800 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fw5j2" Jul 24 17:20:05.085116 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.085094 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-44xrk" Jul 24 17:20:05.093278 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.093255 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r66jf" Jul 24 17:20:05.102344 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.102319 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d9bmx" Jul 24 17:20:05.112539 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.112508 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-499zx" Jul 24 17:20:05.120692 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.120673 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:05.134505 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.134481 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ds7bq" Jul 24 17:20:05.372562 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.372464 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:05.372701 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:05.372608 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:20:05.372701 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:05.372669 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret podName:5f6e8751-f60c-4f88-a369-9158d269f5ff nodeName:}" failed. No retries permitted until 2026-07-24 17:20:06.372653312 +0000 UTC m=+4.024544763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret") pod "global-pull-secret-syncer-7bpxv" (UID: "5f6e8751-f60c-4f88-a369-9158d269f5ff") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:20:05.473118 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.473079 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs\") pod \"network-metrics-daemon-bzsdh\" (UID: \"505f4198-635a-4e18-847d-f85e84b9f70e\") " pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:05.473298 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:05.473255 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:20:05.473352 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:05.473342 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs podName:505f4198-635a-4e18-847d-f85e84b9f70e nodeName:}" failed. No retries permitted until 2026-07-24 17:20:06.473319329 +0000 UTC m=+4.125210784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs") pod "network-metrics-daemon-bzsdh" (UID: "505f4198-635a-4e18-847d-f85e84b9f70e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:20:05.573488 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.573449 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpsbc\" (UniqueName: \"kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc\") pod \"network-check-target-k42w8\" (UID: \"352eb282-95b7-442d-9834-8703f5d0db9a\") " pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:05.573647 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:05.573627 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:20:05.573721 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:05.573658 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:20:05.573721 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:05.573673 2559 projected.go:196] Error preparing data for projected volume kube-api-access-jpsbc for pod openshift-network-diagnostics/network-check-target-k42w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:20:05.573826 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:05.573750 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc podName:352eb282-95b7-442d-9834-8703f5d0db9a nodeName:}" failed. No retries permitted until 2026-07-24 17:20:06.573727456 +0000 UTC m=+4.225618911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jpsbc" (UniqueName: "kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc") pod "network-check-target-k42w8" (UID: "352eb282-95b7-442d-9834-8703f5d0db9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:20:05.811779 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.811657 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 17:15:05 +0000 UTC" deadline="2028-03-31 19:39:46.211688056 +0000 UTC" Jul 24 17:20:05.811779 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:05.811716 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14786h19m40.399975303s" Jul 24 17:20:06.074479 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:06.074439 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067c1ab7_d33d_4d97_a434_4bcb7de77abb.slice/crio-282fa008b243c269c7b27be105e90bbf29d802e932a0ce577c985c550162c207 WatchSource:0}: Error finding container 282fa008b243c269c7b27be105e90bbf29d802e932a0ce577c985c550162c207: Status 404 returned error can't find the container with id 282fa008b243c269c7b27be105e90bbf29d802e932a0ce577c985c550162c207 Jul 24 17:20:06.079825 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:06.079793 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ee13b6_1701_4a39_8160_1a98f36410c3.slice/crio-4fe10eea68a297780e2d6bbb206b0ad614e4aac696f4905086f171d22b665f0f WatchSource:0}: Error finding container 4fe10eea68a297780e2d6bbb206b0ad614e4aac696f4905086f171d22b665f0f: Status 404 returned error can't find the container with id 4fe10eea68a297780e2d6bbb206b0ad614e4aac696f4905086f171d22b665f0f Jul 24 17:20:06.082976 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:06.082948 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a498a6c_63f1_47ec_a1b5_f04d1b2230ff.slice/crio-604d293433bb72912c47d7c94ede9ba5a5ef2c18d49703ceccc33b72c5f22a41 WatchSource:0}: Error finding container 604d293433bb72912c47d7c94ede9ba5a5ef2c18d49703ceccc33b72c5f22a41: Status 404 returned error can't find the container with id 604d293433bb72912c47d7c94ede9ba5a5ef2c18d49703ceccc33b72c5f22a41 Jul 24 17:20:06.100266 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:06.100187 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce67c0eb_7c9f_4c3e_b7a3_3774111e6f7e.slice/crio-9031a95e03c58f05ff97ecba043c3f4f7d8f4a359c4a600229e2209b85d0ec3d WatchSource:0}: Error finding container 9031a95e03c58f05ff97ecba043c3f4f7d8f4a359c4a600229e2209b85d0ec3d: Status 404 returned error can't find the container with id 9031a95e03c58f05ff97ecba043c3f4f7d8f4a359c4a600229e2209b85d0ec3d Jul 24 17:20:06.101693 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:06.101407 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812e91be_8ccb_4e8d_a3d8_185280698b0b.slice/crio-3ea77bf235f20bd6d3ecfc9921e49ef62a1dc555bc1b703a081aeff0083550b0 WatchSource:0}: Error finding container 3ea77bf235f20bd6d3ecfc9921e49ef62a1dc555bc1b703a081aeff0083550b0: Status 404 returned error can't find the container with id 3ea77bf235f20bd6d3ecfc9921e49ef62a1dc555bc1b703a081aeff0083550b0 Jul 24 17:20:06.102415 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:06.102386 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6e0d2b_2415_4e7f_b1c8_c3505f80fe96.slice/crio-37acb3f8eb6a9b161fe34c9025ec72b75cbd9083d23853cc5a19936167dca18a WatchSource:0}: Error finding container 37acb3f8eb6a9b161fe34c9025ec72b75cbd9083d23853cc5a19936167dca18a: Status 404 returned error can't find the container with id 37acb3f8eb6a9b161fe34c9025ec72b75cbd9083d23853cc5a19936167dca18a Jul 24 17:20:06.104543 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:06.103906 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe98c39f_4996_4c21_977f_2545fef7d6b7.slice/crio-639b987f06bd611523a128b3c535bb9e696d47eb9e6bc0b7e3e3998834f996de WatchSource:0}: Error finding container 639b987f06bd611523a128b3c535bb9e696d47eb9e6bc0b7e3e3998834f996de: Status 404 returned error can't find the container with id 639b987f06bd611523a128b3c535bb9e696d47eb9e6bc0b7e3e3998834f996de Jul 24 17:20:06.380017 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.379753 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:06.380206 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:06.379915 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:20:06.380206 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:06.380135 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret podName:5f6e8751-f60c-4f88-a369-9158d269f5ff nodeName:}" failed. No retries permitted until 2026-07-24 17:20:08.380114357 +0000 UTC m=+6.032005809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret") pod "global-pull-secret-syncer-7bpxv" (UID: "5f6e8751-f60c-4f88-a369-9158d269f5ff") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:20:06.481022 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.480980 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs\") pod \"network-metrics-daemon-bzsdh\" (UID: \"505f4198-635a-4e18-847d-f85e84b9f70e\") " pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:06.481195 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:06.481131 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:20:06.481280 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:06.481205 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs podName:505f4198-635a-4e18-847d-f85e84b9f70e nodeName:}" failed. No retries permitted until 2026-07-24 17:20:08.481184112 +0000 UTC m=+6.133075564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs") pod "network-metrics-daemon-bzsdh" (UID: "505f4198-635a-4e18-847d-f85e84b9f70e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:20:06.582123 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.582089 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpsbc\" (UniqueName: \"kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc\") pod \"network-check-target-k42w8\" (UID: \"352eb282-95b7-442d-9834-8703f5d0db9a\") " pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:06.582330 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:06.582296 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:20:06.582330 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:06.582321 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:20:06.582330 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:06.582332 2559 projected.go:196] Error preparing data for projected volume kube-api-access-jpsbc for pod openshift-network-diagnostics/network-check-target-k42w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:20:06.582508 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:06.582395 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc podName:352eb282-95b7-442d-9834-8703f5d0db9a nodeName:}" failed. No retries permitted until 2026-07-24 17:20:08.582379192 +0000 UTC m=+6.234270645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jpsbc" (UniqueName: "kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc") pod "network-check-target-k42w8" (UID: "352eb282-95b7-442d-9834-8703f5d0db9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:20:06.812728 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.812620 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 17:15:05 +0000 UTC" deadline="2028-03-21 17:35:42.278289618 +0000 UTC" Jul 24 17:20:06.812728 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.812671 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14544h15m35.465624065s" Jul 24 17:20:06.941573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.941026 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:06.941573 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:06.941156 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:06.941573 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.941569 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:06.941905 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:06.941702 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:06.941905 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.941772 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:06.941905 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:06.941848 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:06.953765 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.952849 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-105.ec2.internal" event={"ID":"a4b95910892a229c48b40c8bd7c3819e","Type":"ContainerStarted","Data":"32e646d65976a56e97c67d8b1d012d7d0e3b4b277cdd94651b6ebb9116859d37"} Jul 24 17:20:06.958001 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.957919 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66jf" event={"ID":"812e91be-8ccb-4e8d-a3d8-185280698b0b","Type":"ContainerStarted","Data":"3ea77bf235f20bd6d3ecfc9921e49ef62a1dc555bc1b703a081aeff0083550b0"} Jul 24 17:20:06.963900 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.963821 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-44xrk" event={"ID":"ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e","Type":"ContainerStarted","Data":"9031a95e03c58f05ff97ecba043c3f4f7d8f4a359c4a600229e2209b85d0ec3d"} Jul 24 17:20:06.975478 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.975414 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-105.ec2.internal" podStartSLOduration=2.97539214 podStartE2EDuration="2.97539214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:06.97394202 +0000 UTC m=+4.625833498" watchObservedRunningTime="2026-07-24 17:20:06.97539214 +0000 UTC m=+4.627283618" Jul 24 17:20:06.977051 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.976097 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" event={"ID":"31e846fc-447f-4c0a-b40e-a18bb97e7841","Type":"ContainerStarted","Data":"48ab8a034fb62c5853bad823f8b9bc0dece9a3f2332c243f1e61aad7d9fde289"} Jul 24 17:20:06.987503 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.987462 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-499zx" event={"ID":"7a498a6c-63f1-47ec-a1b5-f04d1b2230ff","Type":"ContainerStarted","Data":"604d293433bb72912c47d7c94ede9ba5a5ef2c18d49703ceccc33b72c5f22a41"} Jul 24 17:20:06.997638 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.993286 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" event={"ID":"64ee13b6-1701-4a39-8160-1a98f36410c3","Type":"ContainerStarted","Data":"4fe10eea68a297780e2d6bbb206b0ad614e4aac696f4905086f171d22b665f0f"} Jul 24 17:20:06.997638 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:06.997281 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" event={"ID":"067c1ab7-d33d-4d97-a434-4bcb7de77abb","Type":"ContainerStarted","Data":"282fa008b243c269c7b27be105e90bbf29d802e932a0ce577c985c550162c207"} Jul 24 17:20:07.000930 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:07.000895 2559 generic.go:356] "Generic (PLEG): container finished" podID="ca99be3d06dd468dc82690167e3e51d9" containerID="7c57e036b6b344628f3b77c0b934fab4b1c3c66327e7c823f517f1a1d6004802" exitCode=0 Jul 24 17:20:07.001262 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:07.001197 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal" event={"ID":"ca99be3d06dd468dc82690167e3e51d9","Type":"ContainerDied","Data":"7c57e036b6b344628f3b77c0b934fab4b1c3c66327e7c823f517f1a1d6004802"} Jul 24 17:20:07.005901 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:07.005823 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fw5j2" event={"ID":"fe98c39f-4996-4c21-977f-2545fef7d6b7","Type":"ContainerStarted","Data":"639b987f06bd611523a128b3c535bb9e696d47eb9e6bc0b7e3e3998834f996de"} Jul 24 17:20:07.016770 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:07.016733 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ds7bq" event={"ID":"dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96","Type":"ContainerStarted","Data":"37acb3f8eb6a9b161fe34c9025ec72b75cbd9083d23853cc5a19936167dca18a"} Jul 24 17:20:07.032012 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:07.031972 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d9bmx" event={"ID":"e621415d-ea2d-4520-8353-752e48ab5b39","Type":"ContainerStarted","Data":"f0d82e6b3359ba63594beb63399179e6b26fd0f8e404570c8f798f4c37767908"} Jul 24 17:20:08.054630 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:08.054443 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal" event={"ID":"ca99be3d06dd468dc82690167e3e51d9","Type":"ContainerStarted","Data":"32cef10d33b8927f31e8e8426cb3e3f7c95b93371f03ce066a8b1781130cf0eb"} Jul 24 17:20:08.073303 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:08.073240 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-105.ec2.internal" podStartSLOduration=4.073216805 podStartE2EDuration="4.073216805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:08.073074554 +0000 UTC m=+5.724966030" watchObservedRunningTime="2026-07-24 17:20:08.073216805 +0000 UTC m=+5.725108283" Jul 24 17:20:08.396613 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:08.396498 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:08.396799 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:08.396715 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:20:08.396799 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:08.396784 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret podName:5f6e8751-f60c-4f88-a369-9158d269f5ff nodeName:}" failed. No retries permitted until 2026-07-24 17:20:12.396765049 +0000 UTC m=+10.048656508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret") pod "global-pull-secret-syncer-7bpxv" (UID: "5f6e8751-f60c-4f88-a369-9158d269f5ff") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:20:08.497620 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:08.497556 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs\") pod \"network-metrics-daemon-bzsdh\" (UID: \"505f4198-635a-4e18-847d-f85e84b9f70e\") " pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:08.497789 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:08.497757 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:20:08.497830 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:08.497824 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs podName:505f4198-635a-4e18-847d-f85e84b9f70e nodeName:}" failed. No retries permitted until 2026-07-24 17:20:12.497804769 +0000 UTC m=+10.149696224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs") pod "network-metrics-daemon-bzsdh" (UID: "505f4198-635a-4e18-847d-f85e84b9f70e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:20:08.599213 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:08.598555 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpsbc\" (UniqueName: \"kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc\") pod \"network-check-target-k42w8\" (UID: \"352eb282-95b7-442d-9834-8703f5d0db9a\") " pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:08.599213 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:08.598726 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:20:08.599213 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:08.598749 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:20:08.599213 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:08.598763 2559 projected.go:196] Error preparing data for projected volume kube-api-access-jpsbc for pod openshift-network-diagnostics/network-check-target-k42w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:20:08.599213 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:08.598826 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc podName:352eb282-95b7-442d-9834-8703f5d0db9a nodeName:}" failed. No retries permitted until 2026-07-24 17:20:12.598807826 +0000 UTC m=+10.250699281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jpsbc" (UniqueName: "kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc") pod "network-check-target-k42w8" (UID: "352eb282-95b7-442d-9834-8703f5d0db9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:20:08.937716 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:08.937684 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:08.937716 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:08.937667 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:08.937957 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:08.937842 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:08.938491 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:08.938290 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:08.938491 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:08.938352 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:08.938491 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:08.938427 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:10.937692 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:10.937624 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:10.938139 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:10.937740 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:10.938139 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:10.937753 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:10.938139 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:10.937633 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:10.938139 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:10.937849 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:10.938139 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:10.937929 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:12.432146 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:12.432101 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:12.432677 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:12.432316 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:20:12.432677 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:12.432378 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret podName:5f6e8751-f60c-4f88-a369-9158d269f5ff nodeName:}" failed. No retries permitted until 2026-07-24 17:20:20.432359177 +0000 UTC m=+18.084250635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret") pod "global-pull-secret-syncer-7bpxv" (UID: "5f6e8751-f60c-4f88-a369-9158d269f5ff") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:20:12.533235 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:12.533191 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs\") pod \"network-metrics-daemon-bzsdh\" (UID: \"505f4198-635a-4e18-847d-f85e84b9f70e\") " pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:12.533423 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:12.533374 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:20:12.533483 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:12.533444 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs podName:505f4198-635a-4e18-847d-f85e84b9f70e nodeName:}" failed. No retries permitted until 2026-07-24 17:20:20.533424407 +0000 UTC m=+18.185315858 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs") pod "network-metrics-daemon-bzsdh" (UID: "505f4198-635a-4e18-847d-f85e84b9f70e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:20:12.634098 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:12.634059 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpsbc\" (UniqueName: \"kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc\") pod \"network-check-target-k42w8\" (UID: \"352eb282-95b7-442d-9834-8703f5d0db9a\") " pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:12.634277 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:12.634259 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:20:12.634336 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:12.634288 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:20:12.634336 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:12.634301 2559 projected.go:196] Error preparing data for projected volume kube-api-access-jpsbc for pod openshift-network-diagnostics/network-check-target-k42w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:20:12.634453 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:12.634361 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc podName:352eb282-95b7-442d-9834-8703f5d0db9a nodeName:}" failed. No retries permitted until 2026-07-24 17:20:20.634341982 +0000 UTC m=+18.286233456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jpsbc" (UniqueName: "kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc") pod "network-check-target-k42w8" (UID: "352eb282-95b7-442d-9834-8703f5d0db9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:20:12.937908 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:12.937870 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:12.938071 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:12.937987 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:12.938396 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:12.938381 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:12.938484 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:12.938466 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:12.938597 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:12.938583 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:12.938711 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:12.938684 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:14.937655 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:14.937617 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:14.937655 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:14.937633 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:14.938190 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:14.937749 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:14.938190 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:14.937855 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:14.938190 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:14.937913 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:14.938190 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:14.937989 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:16.937978 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:16.937938 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:16.938389 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:16.937938 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:16.938389 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:16.938050 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:16.938389 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:16.938068 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:16.938389 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:16.938159 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:16.938389 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:16.938236 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:18.937533 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:18.937493 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:18.938053 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:18.937555 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:18.938053 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:18.937493 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:18.938053 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:18.937644 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:18.938053 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:18.937718 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:18.938053 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:18.937820 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:20.492133 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:20.492092 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:20.492643 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:20.492265 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:20:20.492643 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:20.492338 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret podName:5f6e8751-f60c-4f88-a369-9158d269f5ff nodeName:}" failed. No retries permitted until 2026-07-24 17:20:36.492317763 +0000 UTC m=+34.144209221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret") pod "global-pull-secret-syncer-7bpxv" (UID: "5f6e8751-f60c-4f88-a369-9158d269f5ff") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:20:20.592777 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:20.592740 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs\") pod \"network-metrics-daemon-bzsdh\" (UID: \"505f4198-635a-4e18-847d-f85e84b9f70e\") " pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:20.592959 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:20.592900 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:20:20.593035 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:20.592982 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs podName:505f4198-635a-4e18-847d-f85e84b9f70e nodeName:}" failed. No retries permitted until 2026-07-24 17:20:36.592959724 +0000 UTC m=+34.244851184 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs") pod "network-metrics-daemon-bzsdh" (UID: "505f4198-635a-4e18-847d-f85e84b9f70e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:20:20.694064 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:20.694028 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpsbc\" (UniqueName: \"kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc\") pod \"network-check-target-k42w8\" (UID: \"352eb282-95b7-442d-9834-8703f5d0db9a\") " pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:20.694256 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:20.694201 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:20:20.694256 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:20.694223 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:20:20.694256 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:20.694234 2559 projected.go:196] Error preparing data for projected volume kube-api-access-jpsbc for pod openshift-network-diagnostics/network-check-target-k42w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:20:20.694464 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:20.694298 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc podName:352eb282-95b7-442d-9834-8703f5d0db9a nodeName:}" failed. No retries permitted until 2026-07-24 17:20:36.694279164 +0000 UTC m=+34.346170622 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jpsbc" (UniqueName: "kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc") pod "network-check-target-k42w8" (UID: "352eb282-95b7-442d-9834-8703f5d0db9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:20:20.938041 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:20.937967 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:20.938181 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:20.937971 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:20.938181 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:20.938067 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:20.938271 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:20.938248 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:20.938314 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:20.938285 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:20.938376 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:20.938359 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:22.939949 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:22.939920 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:22.939949 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:22.939936 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:22.939949 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:22.939953 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:22.940545 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:22.940028 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:22.940545 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:22.940100 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:22.940545 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:22.940167 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:24.937976 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:24.937639 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:24.937976 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:24.937770 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:24.937976 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:24.937845 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:24.937976 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:24.937911 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:24.938731 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:24.938581 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:24.938731 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:24.938687 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:25.093129 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:25.092716 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fw5j2" event={"ID":"fe98c39f-4996-4c21-977f-2545fef7d6b7","Type":"ContainerStarted","Data":"37e712b6d25bd7f8181797c2767de3579deb87e823f057a6d98b3aadb28a9e93"} Jul 24 17:20:25.095786 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:25.095747 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ds7bq" event={"ID":"dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96","Type":"ContainerStarted","Data":"dee68e1052e8d5676d0c3a6027f10d373f3459773e4a70ed54bf5da57c20c57b"} Jul 24 17:20:25.097454 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:25.097428 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d9bmx" event={"ID":"e621415d-ea2d-4520-8353-752e48ab5b39","Type":"ContainerStarted","Data":"e5f8315de0b5a6f726798d02df72f41240f2b09a51f95147214a221b6e4a7e58"} Jul 24 17:20:25.098914 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:25.098873 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66jf" event={"ID":"812e91be-8ccb-4e8d-a3d8-185280698b0b","Type":"ContainerStarted","Data":"ecfa9b64df138e08f869a06baf123104a7a032171b3c960962246061686bc00a"} Jul 24 17:20:25.100831 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:25.100752 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" event={"ID":"31e846fc-447f-4c0a-b40e-a18bb97e7841","Type":"ContainerStarted","Data":"e874fe55a8b044deeaf3aa3d9220d1d2b3be83245b813038c982ab7393c39bf1"} Jul 24 17:20:25.102123 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:25.102084 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-499zx" event={"ID":"7a498a6c-63f1-47ec-a1b5-f04d1b2230ff","Type":"ContainerStarted","Data":"86a2ac131cb42ffb00e3b4aacc850a520e68f5f20c0fb4746144b2d6ee84bbd2"} Jul 24 17:20:25.103766 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:25.103737 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" event={"ID":"64ee13b6-1701-4a39-8160-1a98f36410c3","Type":"ContainerStarted","Data":"7175d9ae5e26d597f7683d326459d1eb0f433c8cebef46eb860ff57577a2584b"} Jul 24 17:20:25.106657 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:25.106629 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" event={"ID":"067c1ab7-d33d-4d97-a434-4bcb7de77abb","Type":"ContainerStarted","Data":"96e5d4890074aa3707b6df112662c148a31d9603a5dd2f494f35737a765efd37"} Jul 24 17:20:25.141993 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:25.141807 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cwrz4" podStartSLOduration=3.328657166 podStartE2EDuration="22.141790605s" totalImagesPullingTime="18.813133439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:25.141278363 +0000 UTC m=+22.793169836" watchObservedRunningTime="2026-07-24 17:20:25.141790605 +0000 UTC m=+22.793682077" Jul 24 17:20:25.142125 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:25.142105 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fw5j2" podStartSLOduration=3.315277034 podStartE2EDuration="22.142098964s" totalImagesPullingTime="18.82682193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:25.115617047 +0000 UTC m=+22.767508520" watchObservedRunningTime="2026-07-24 17:20:25.142098964 +0000 UTC m=+22.793990437" Jul 24 17:20:25.214033 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:25.213985 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-499zx" podStartSLOduration=3.439438959 podStartE2EDuration="22.213969567s" totalImagesPullingTime="18.774530608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:25.213928122 +0000 UTC m=+22.865819599" watchObservedRunningTime="2026-07-24 17:20:25.213969567 +0000 UTC m=+22.865861039" Jul 24 17:20:25.231327 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:25.231274 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d9bmx" podStartSLOduration=11.200826631 podStartE2EDuration="22.231258651s" totalImagesPullingTime="11.03043202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:25.231164372 +0000 UTC m=+22.883055882" watchObservedRunningTime="2026-07-24 17:20:25.231258651 +0000 UTC m=+22.883150123" Jul 24 17:20:25.250535 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:25.250478 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ds7bq" podStartSLOduration=3.478753293 podStartE2EDuration="22.250462979s" totalImagesPullingTime="18.771709686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:25.249402856 +0000 UTC m=+22.901294329" watchObservedRunningTime="2026-07-24 17:20:25.250462979 +0000 UTC m=+22.902354451" Jul 24 17:20:26.111212 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:26.111180 2559 generic.go:356] "Generic (PLEG): container finished" podID="812e91be-8ccb-4e8d-a3d8-185280698b0b" containerID="ecfa9b64df138e08f869a06baf123104a7a032171b3c960962246061686bc00a" exitCode=0 Jul 24 17:20:26.112026 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:26.111268 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66jf" event={"ID":"812e91be-8ccb-4e8d-a3d8-185280698b0b","Type":"ContainerDied","Data":"ecfa9b64df138e08f869a06baf123104a7a032171b3c960962246061686bc00a"} Jul 24 17:20:26.115093 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:26.115071 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" event={"ID":"31e846fc-447f-4c0a-b40e-a18bb97e7841","Type":"ContainerStarted","Data":"9a92a4452237a9818f0a4bbadcd9b6638c47b9c7717f874b4201ae486cf3de0b"} Jul 24 17:20:26.115199 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:26.115150 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" event={"ID":"31e846fc-447f-4c0a-b40e-a18bb97e7841","Type":"ContainerStarted","Data":"119d867ebae1e51fcb8cc29560bbbced9522cec6a18976803c25c34ae22531c0"} Jul 24 17:20:26.115269 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:26.115209 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" event={"ID":"31e846fc-447f-4c0a-b40e-a18bb97e7841","Type":"ContainerStarted","Data":"4efb1f9d3d27adc73a8712d446c4ebb759e704c5ddfab1eccaef1b6042c6f10e"} Jul 24 17:20:26.115269 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:26.115259 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" event={"ID":"31e846fc-447f-4c0a-b40e-a18bb97e7841","Type":"ContainerStarted","Data":"bc66a9dfe1f888c13052b4f4f7904ed9d3cce0df5a65e029ac2d27ac45b32685"} Jul 24 17:20:26.115365 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:26.115307 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" event={"ID":"31e846fc-447f-4c0a-b40e-a18bb97e7841","Type":"ContainerStarted","Data":"e25ce44ed63f69527da8741fd2ef3d0d5e2ac45c2d85f087a318300e3a443070"} Jul 24 17:20:26.587845 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:26.587819 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 17:20:26.830675 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:26.830526 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T17:20:26.587839289Z","UUID":"5e4f87a3-e978-4c04-a6d4-118a06be99ef","Handler":null,"Name":"","Endpoint":""} Jul 24 17:20:26.832491 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:26.832467 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 17:20:26.832631 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:26.832505 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 17:20:26.940892 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:26.940868 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:26.941078 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:26.940868 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:26.941078 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:26.940984 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:26.941078 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:26.940879 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:26.941078 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:26.941048 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:26.941285 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:26.941131 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:27.119719 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:27.119633 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" event={"ID":"067c1ab7-d33d-4d97-a434-4bcb7de77abb","Type":"ContainerStarted","Data":"7fdbc0cfc26e739069994cfcc21515a051740c4f0ecd063ca63bae108e66ad95"} Jul 24 17:20:27.121413 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:27.121382 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-44xrk" event={"ID":"ce67c0eb-7c9f-4c3e-b7a3-3774111e6f7e","Type":"ContainerStarted","Data":"9acfa8bbfc3089620cd307bb02dcd6d10a26a1e366b0fea3a03adfe33d7f201b"} Jul 24 17:20:27.141616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:27.141567 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-44xrk" podStartSLOduration=5.369840546 podStartE2EDuration="24.14155082s" totalImagesPullingTime="18.771710274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:27.140797706 +0000 UTC m=+24.792689179" watchObservedRunningTime="2026-07-24 17:20:27.14155082 +0000 UTC m=+24.793442294" Jul 24 17:20:28.100466 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:28.100426 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d9bmx" Jul 24 17:20:28.101022 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:28.101006 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d9bmx" Jul 24 17:20:28.127267 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:28.127225 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" event={"ID":"31e846fc-447f-4c0a-b40e-a18bb97e7841","Type":"ContainerStarted","Data":"4d01bdac80483e6b55694d68792585ffa70f61210eff869472adfbbbad9343a5"} Jul 24 17:20:28.127732 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:28.127552 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d9bmx" Jul 24 17:20:28.128119 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:28.128099 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d9bmx" Jul 24 17:20:28.941466 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:28.941440 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:28.941466 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:28.941479 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:28.941790 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:28.941482 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:28.941790 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:28.941604 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:28.941790 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:28.941679 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:28.941790 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:28.941766 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:29.132560 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:29.132304 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" event={"ID":"067c1ab7-d33d-4d97-a434-4bcb7de77abb","Type":"ContainerStarted","Data":"b3aa85330eb88f1cf2099ff239a6c3d574d7e7fbf3ff2662598d0af439aa936f"} Jul 24 17:20:29.164847 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:29.164785 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-96wlq" podStartSLOduration=4.552253942 podStartE2EDuration="26.164761976s" totalImagesPullingTime="21.612508034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:20:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:29.163597085 +0000 UTC m=+26.815488549" watchObservedRunningTime="2026-07-24 17:20:29.164761976 +0000 UTC m=+26.816653450" Jul 24 17:20:29.843533 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:29.843480 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ds7bq_dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96/dns-node-resolver/0.log" Jul 24 17:20:30.408578 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:30.408554 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-499zx_7a498a6c-63f1-47ec-a1b5-f04d1b2230ff/node-ca/0.log" Jul 24 17:20:30.941375 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:30.941115 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:30.941505 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:30.941112 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:30.941505 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:30.941148 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:30.941633 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:30.941545 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:30.941692 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:30.941628 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:30.941744 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:30.941727 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:31.139494 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:31.139462 2559 generic.go:356] "Generic (PLEG): container finished" podID="812e91be-8ccb-4e8d-a3d8-185280698b0b" containerID="244608820e67ad30ffa611fa9d4613d8456f112fd39c4b854fa5671a4e3f2134" exitCode=0 Jul 24 17:20:31.139667 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:31.139565 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66jf" event={"ID":"812e91be-8ccb-4e8d-a3d8-185280698b0b","Type":"ContainerDied","Data":"244608820e67ad30ffa611fa9d4613d8456f112fd39c4b854fa5671a4e3f2134"} Jul 24 17:20:31.145399 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:31.145377 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" event={"ID":"31e846fc-447f-4c0a-b40e-a18bb97e7841","Type":"ContainerStarted","Data":"c31a1b8f54e741fafa507879bf98767870f8b535ea3e1b4345a5135dd5883fa5"} Jul 24 17:20:31.145709 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:31.145693 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:31.161237 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:31.161207 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:31.190000 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:31.189893 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" podStartSLOduration=9.348841045 podStartE2EDuration="28.189876862s" totalImagesPullingTime="18.841035817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:31.189715686 +0000 UTC m=+28.841607158" watchObservedRunningTime="2026-07-24 17:20:31.189876862 +0000 UTC m=+28.841768337" Jul 24 17:20:32.148553 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:32.148507 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:32.149285 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:32.148616 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:32.163848 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:32.163824 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:20:32.818506 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:32.818409 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k42w8"] Jul 24 17:20:32.818685 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:32.818592 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:32.818746 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:32.818708 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:32.820621 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:32.820592 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bzsdh"] Jul 24 17:20:32.820757 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:32.820712 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:32.820834 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:32.820814 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:32.824141 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:32.824111 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7bpxv"] Jul 24 17:20:32.824256 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:32.824205 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:32.824319 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:32.824289 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:33.152019 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:33.151934 2559 generic.go:356] "Generic (PLEG): container finished" podID="812e91be-8ccb-4e8d-a3d8-185280698b0b" containerID="d4e9f1f235b396fbc3f5579d384e34db57dc8b2809b76da102442318bb9db432" exitCode=0 Jul 24 17:20:33.152432 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:33.152022 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66jf" event={"ID":"812e91be-8ccb-4e8d-a3d8-185280698b0b","Type":"ContainerDied","Data":"d4e9f1f235b396fbc3f5579d384e34db57dc8b2809b76da102442318bb9db432"} Jul 24 17:20:33.937182 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:33.937158 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:33.937289 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:33.937272 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:34.938107 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:34.937930 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:34.938479 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:34.937985 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:34.938479 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:34.938187 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:34.938479 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:34.938275 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:35.158413 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:35.158327 2559 generic.go:356] "Generic (PLEG): container finished" podID="812e91be-8ccb-4e8d-a3d8-185280698b0b" containerID="3c8163f88d098968972b62f92fa8530ce54a6972984b04a647fecc88df7a1fcc" exitCode=0 Jul 24 17:20:35.158413 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:35.158366 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66jf" event={"ID":"812e91be-8ccb-4e8d-a3d8-185280698b0b","Type":"ContainerDied","Data":"3c8163f88d098968972b62f92fa8530ce54a6972984b04a647fecc88df7a1fcc"} Jul 24 17:20:35.937589 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:35.937556 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:35.937814 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:35.937668 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:36.510128 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:36.510064 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:36.510498 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:36.510212 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:20:36.510498 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:36.510276 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret podName:5f6e8751-f60c-4f88-a369-9158d269f5ff nodeName:}" failed. No retries permitted until 2026-07-24 17:21:08.510260834 +0000 UTC m=+66.162152285 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret") pod "global-pull-secret-syncer-7bpxv" (UID: "5f6e8751-f60c-4f88-a369-9158d269f5ff") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:20:36.611343 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:36.611308 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs\") pod \"network-metrics-daemon-bzsdh\" (UID: \"505f4198-635a-4e18-847d-f85e84b9f70e\") " pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:36.611505 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:36.611481 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:20:36.611597 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:36.611583 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs podName:505f4198-635a-4e18-847d-f85e84b9f70e nodeName:}" failed. No retries permitted until 2026-07-24 17:21:08.611563218 +0000 UTC m=+66.263454680 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs") pod "network-metrics-daemon-bzsdh" (UID: "505f4198-635a-4e18-847d-f85e84b9f70e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:20:36.712555 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:36.712499 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpsbc\" (UniqueName: \"kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc\") pod \"network-check-target-k42w8\" (UID: \"352eb282-95b7-442d-9834-8703f5d0db9a\") " pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:36.712723 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:36.712700 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:20:36.712767 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:36.712727 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:20:36.712767 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:36.712738 2559 projected.go:196] Error preparing data for projected volume kube-api-access-jpsbc for pod openshift-network-diagnostics/network-check-target-k42w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:20:36.712838 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:36.712788 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc podName:352eb282-95b7-442d-9834-8703f5d0db9a nodeName:}" failed. No retries permitted until 2026-07-24 17:21:08.712775736 +0000 UTC m=+66.364667190 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jpsbc" (UniqueName: "kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc") pod "network-check-target-k42w8" (UID: "352eb282-95b7-442d-9834-8703f5d0db9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:20:36.938034 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:36.937956 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:36.938199 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:36.937956 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:36.938199 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:36.938111 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzsdh" podUID="505f4198-635a-4e18-847d-f85e84b9f70e" Jul 24 17:20:36.938199 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:36.938164 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k42w8" podUID="352eb282-95b7-442d-9834-8703f5d0db9a" Jul 24 17:20:37.937901 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:37.937868 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:37.938377 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:37.937993 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7bpxv" podUID="5f6e8751-f60c-4f88-a369-9158d269f5ff" Jul 24 17:20:38.625258 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.625227 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-105.ec2.internal" event="NodeReady" Jul 24 17:20:38.625430 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.625365 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 17:20:38.678570 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.678469 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5fcw2"] Jul 24 17:20:38.681076 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.681049 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5fcw2" Jul 24 17:20:38.681357 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.681335 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ptzgn"] Jul 24 17:20:38.684508 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.684051 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 17:20:38.684508 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.684176 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 17:20:38.684508 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.684342 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pkrw7\"" Jul 24 17:20:38.684508 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.684457 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 17:20:38.686243 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.686224 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.690394 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.689752 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 17:20:38.690394 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.689767 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ts8s2\"" Jul 24 17:20:38.690394 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.690000 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 17:20:38.690394 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.690002 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 17:20:38.690394 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.690359 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 17:20:38.692124 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.692104 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5fcw2"] Jul 24 17:20:38.698242 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.698223 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wxkxq"] Jul 24 17:20:38.700017 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.699995 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ptzgn"] Jul 24 17:20:38.700119 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.700105 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:38.702727 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.702710 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 17:20:38.702809 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.702762 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 17:20:38.702936 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.702918 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jtfqv\"" Jul 24 17:20:38.715361 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.715337 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wxkxq"] Jul 24 17:20:38.825693 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.825654 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.825861 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.825718 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f49740-2957-426b-9a28-a1f915692dd4-tmp-dir\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:38.825861 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.825820 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f49740-2957-426b-9a28-a1f915692dd4-config-volume\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:38.825861 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.825852 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.826034 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.825896 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-crio-socket\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.826034 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.825922 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vndmc\" (UniqueName: \"kubernetes.io/projected/65334c29-2f2c-4b2f-8e30-acf551b4355a-kube-api-access-vndmc\") pod \"ingress-canary-5fcw2\" (UID: \"65334c29-2f2c-4b2f-8e30-acf551b4355a\") " pod="openshift-ingress-canary/ingress-canary-5fcw2" Jul 24 17:20:38.826034 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.825956 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-data-volume\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.826034 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.826010 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hjx7\" (UniqueName: \"kubernetes.io/projected/f4f49740-2957-426b-9a28-a1f915692dd4-kube-api-access-4hjx7\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:38.826208 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.826032 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65334c29-2f2c-4b2f-8e30-acf551b4355a-cert\") pod \"ingress-canary-5fcw2\" (UID: \"65334c29-2f2c-4b2f-8e30-acf551b4355a\") " pod="openshift-ingress-canary/ingress-canary-5fcw2" Jul 24 17:20:38.826208 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.826060 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9697j\" (UniqueName: \"kubernetes.io/projected/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-kube-api-access-9697j\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.826208 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.826090 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:38.927279 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.927240 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.927279 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.927291 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f49740-2957-426b-9a28-a1f915692dd4-tmp-dir\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:38.927568 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.927339 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f49740-2957-426b-9a28-a1f915692dd4-config-volume\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:38.927568 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.927365 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.927568 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.927399 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-crio-socket\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.927568 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:38.927415 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:20:38.927568 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:38.927551 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls podName:c7ca7e6f-0549-47c4-ba36-7195702ca2fd nodeName:}" failed. No retries permitted until 2026-07-24 17:20:39.427529124 +0000 UTC m=+37.079420590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ptzgn" (UID: "c7ca7e6f-0549-47c4-ba36-7195702ca2fd") : secret "insights-runtime-extractor-tls" not found Jul 24 17:20:38.927829 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.927424 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndmc\" (UniqueName: \"kubernetes.io/projected/65334c29-2f2c-4b2f-8e30-acf551b4355a-kube-api-access-vndmc\") pod \"ingress-canary-5fcw2\" (UID: \"65334c29-2f2c-4b2f-8e30-acf551b4355a\") " pod="openshift-ingress-canary/ingress-canary-5fcw2" Jul 24 17:20:38.927829 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.927718 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f49740-2957-426b-9a28-a1f915692dd4-tmp-dir\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:38.927829 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.927710 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-data-volume\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.927829 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.927815 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hjx7\" (UniqueName: \"kubernetes.io/projected/f4f49740-2957-426b-9a28-a1f915692dd4-kube-api-access-4hjx7\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:38.928027 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.927840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65334c29-2f2c-4b2f-8e30-acf551b4355a-cert\") pod \"ingress-canary-5fcw2\" (UID: \"65334c29-2f2c-4b2f-8e30-acf551b4355a\") " pod="openshift-ingress-canary/ingress-canary-5fcw2" Jul 24 17:20:38.928027 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.927869 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9697j\" (UniqueName: \"kubernetes.io/projected/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-kube-api-access-9697j\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.928027 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.927890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:38.928027 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:38.927980 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:20:38.928219 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:38.928031 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls podName:f4f49740-2957-426b-9a28-a1f915692dd4 nodeName:}" failed. No retries permitted until 2026-07-24 17:20:39.428015837 +0000 UTC m=+37.079907289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls") pod "dns-default-wxkxq" (UID: "f4f49740-2957-426b-9a28-a1f915692dd4") : secret "dns-default-metrics-tls" not found Jul 24 17:20:38.928219 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.928186 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f49740-2957-426b-9a28-a1f915692dd4-config-volume\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:38.928388 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.927981 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-data-volume\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.928443 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.928385 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.928643 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.928599 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-crio-socket\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.932831 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.932805 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65334c29-2f2c-4b2f-8e30-acf551b4355a-cert\") pod \"ingress-canary-5fcw2\" (UID: \"65334c29-2f2c-4b2f-8e30-acf551b4355a\") " pod="openshift-ingress-canary/ingress-canary-5fcw2" Jul 24 17:20:38.938424 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.938402 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:20:38.939013 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.938776 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:20:38.957202 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.957173 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 17:20:38.957348 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.957254 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8rjf6\"" Jul 24 17:20:38.957985 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.957799 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 17:20:38.958438 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.958413 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 17:20:38.958665 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.958642 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jwvtx\"" Jul 24 17:20:38.960039 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.960019 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hjx7\" (UniqueName: \"kubernetes.io/projected/f4f49740-2957-426b-9a28-a1f915692dd4-kube-api-access-4hjx7\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:38.960182 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.960168 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9697j\" (UniqueName: \"kubernetes.io/projected/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-kube-api-access-9697j\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:38.960260 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.960223 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vndmc\" (UniqueName: \"kubernetes.io/projected/65334c29-2f2c-4b2f-8e30-acf551b4355a-kube-api-access-vndmc\") pod \"ingress-canary-5fcw2\" (UID: \"65334c29-2f2c-4b2f-8e30-acf551b4355a\") " pod="openshift-ingress-canary/ingress-canary-5fcw2" Jul 24 17:20:38.994670 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:38.994603 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5fcw2" Jul 24 17:20:39.433108 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:39.433080 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:39.433225 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:39.433126 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:39.433282 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:39.433260 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:20:39.433282 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:39.433263 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:20:39.433498 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:39.433332 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls podName:f4f49740-2957-426b-9a28-a1f915692dd4 nodeName:}" failed. No retries permitted until 2026-07-24 17:20:40.433309627 +0000 UTC m=+38.085201081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls") pod "dns-default-wxkxq" (UID: "f4f49740-2957-426b-9a28-a1f915692dd4") : secret "dns-default-metrics-tls" not found Jul 24 17:20:39.433498 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:39.433353 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls podName:c7ca7e6f-0549-47c4-ba36-7195702ca2fd nodeName:}" failed. No retries permitted until 2026-07-24 17:20:40.433341942 +0000 UTC m=+38.085233394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ptzgn" (UID: "c7ca7e6f-0549-47c4-ba36-7195702ca2fd") : secret "insights-runtime-extractor-tls" not found Jul 24 17:20:39.533469 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:39.533273 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5fcw2"] Jul 24 17:20:39.536369 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:39.536338 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65334c29_2f2c_4b2f_8e30_acf551b4355a.slice/crio-1690739743aca5afc17ba07173c5034d42717530be2a3a466b29d30522691ef2 WatchSource:0}: Error finding container 1690739743aca5afc17ba07173c5034d42717530be2a3a466b29d30522691ef2: Status 404 returned error can't find the container with id 1690739743aca5afc17ba07173c5034d42717530be2a3a466b29d30522691ef2 Jul 24 17:20:39.937485 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:39.937456 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:20:39.940448 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:39.940431 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 17:20:40.173602 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:40.173563 2559 generic.go:356] "Generic (PLEG): container finished" podID="812e91be-8ccb-4e8d-a3d8-185280698b0b" containerID="c4e7c678b4af183611345411906378aaaaf272876add1b4462fbea3e42f1005b" exitCode=0 Jul 24 17:20:40.173759 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:40.173641 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66jf" event={"ID":"812e91be-8ccb-4e8d-a3d8-185280698b0b","Type":"ContainerDied","Data":"c4e7c678b4af183611345411906378aaaaf272876add1b4462fbea3e42f1005b"} Jul 24 17:20:40.174833 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:40.174800 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5fcw2" event={"ID":"65334c29-2f2c-4b2f-8e30-acf551b4355a","Type":"ContainerStarted","Data":"1690739743aca5afc17ba07173c5034d42717530be2a3a466b29d30522691ef2"} Jul 24 17:20:40.446630 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:40.446502 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:40.446630 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:40.446594 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:40.446820 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:40.446642 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:20:40.446820 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:40.446724 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls podName:f4f49740-2957-426b-9a28-a1f915692dd4 nodeName:}" failed. No retries permitted until 2026-07-24 17:20:42.446702679 +0000 UTC m=+40.098594133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls") pod "dns-default-wxkxq" (UID: "f4f49740-2957-426b-9a28-a1f915692dd4") : secret "dns-default-metrics-tls" not found Jul 24 17:20:40.446820 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:40.446736 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:20:40.446820 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:40.446792 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls podName:c7ca7e6f-0549-47c4-ba36-7195702ca2fd nodeName:}" failed. No retries permitted until 2026-07-24 17:20:42.446779125 +0000 UTC m=+40.098670576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ptzgn" (UID: "c7ca7e6f-0549-47c4-ba36-7195702ca2fd") : secret "insights-runtime-extractor-tls" not found Jul 24 17:20:41.180850 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:41.180818 2559 generic.go:356] "Generic (PLEG): container finished" podID="812e91be-8ccb-4e8d-a3d8-185280698b0b" containerID="9b2b4b2c9fe639aba148d7ab7133e7b36940ffca6af09d1678126a6f975e0084" exitCode=0 Jul 24 17:20:41.181238 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:41.180879 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66jf" event={"ID":"812e91be-8ccb-4e8d-a3d8-185280698b0b","Type":"ContainerDied","Data":"9b2b4b2c9fe639aba148d7ab7133e7b36940ffca6af09d1678126a6f975e0084"} Jul 24 17:20:42.186137 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:42.186102 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66jf" event={"ID":"812e91be-8ccb-4e8d-a3d8-185280698b0b","Type":"ContainerStarted","Data":"8ac806aa1fc0865b704777363aa368e245f25a79276b51f78eab3e435b196fc9"} Jul 24 17:20:42.208807 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:42.208754 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r66jf" podStartSLOduration=9.292030941 podStartE2EDuration="39.20873951s" totalImagesPullingTime="29.916708569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:20:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:42.207022277 +0000 UTC m=+39.858913750" watchObservedRunningTime="2026-07-24 17:20:42.20873951 +0000 UTC m=+39.860630982" Jul 24 17:20:42.464822 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:42.464720 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:42.464822 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:42.464823 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:42.465024 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:42.464889 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:20:42.465024 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:42.464901 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:20:42.465024 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:42.464955 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls podName:f4f49740-2957-426b-9a28-a1f915692dd4 nodeName:}" failed. No retries permitted until 2026-07-24 17:20:46.464943347 +0000 UTC m=+44.116834797 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls") pod "dns-default-wxkxq" (UID: "f4f49740-2957-426b-9a28-a1f915692dd4") : secret "dns-default-metrics-tls" not found Jul 24 17:20:42.465024 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:42.464969 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls podName:c7ca7e6f-0549-47c4-ba36-7195702ca2fd nodeName:}" failed. No retries permitted until 2026-07-24 17:20:46.464963015 +0000 UTC m=+44.116854466 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ptzgn" (UID: "c7ca7e6f-0549-47c4-ba36-7195702ca2fd") : secret "insights-runtime-extractor-tls" not found Jul 24 17:20:43.189410 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:43.189374 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5fcw2" event={"ID":"65334c29-2f2c-4b2f-8e30-acf551b4355a","Type":"ContainerStarted","Data":"2584630dde1cea8f0955e869655406466cf809666347509a0c72a581ee104f45"} Jul 24 17:20:43.209867 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:43.209820 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5fcw2" podStartSLOduration=2.41875892 podStartE2EDuration="5.209804342s" totalImagesPullingTime="2.791045422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:43.208683478 +0000 UTC m=+40.860574950" watchObservedRunningTime="2026-07-24 17:20:43.209804342 +0000 UTC m=+40.861695815" Jul 24 17:20:46.492399 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:46.492356 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:46.492399 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:46.492398 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:46.492849 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:46.492507 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:20:46.492849 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:46.492527 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:20:46.492849 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:46.492579 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls podName:c7ca7e6f-0549-47c4-ba36-7195702ca2fd nodeName:}" failed. No retries permitted until 2026-07-24 17:20:54.49256522 +0000 UTC m=+52.144456671 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ptzgn" (UID: "c7ca7e6f-0549-47c4-ba36-7195702ca2fd") : secret "insights-runtime-extractor-tls" not found Jul 24 17:20:46.492849 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:20:46.492599 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls podName:f4f49740-2957-426b-9a28-a1f915692dd4 nodeName:}" failed. No retries permitted until 2026-07-24 17:20:54.492585233 +0000 UTC m=+52.144476686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls") pod "dns-default-wxkxq" (UID: "f4f49740-2957-426b-9a28-a1f915692dd4") : secret "dns-default-metrics-tls" not found Jul 24 17:20:54.551577 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:54.551536 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:54.551577 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:54.551579 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:54.555248 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:54.555222 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f49740-2957-426b-9a28-a1f915692dd4-metrics-tls\") pod \"dns-default-wxkxq\" (UID: \"f4f49740-2957-426b-9a28-a1f915692dd4\") " pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:54.555248 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:54.555243 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7ca7e6f-0549-47c4-ba36-7195702ca2fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptzgn\" (UID: \"c7ca7e6f-0549-47c4-ba36-7195702ca2fd\") " pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:54.601324 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:54.601291 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ptzgn" Jul 24 17:20:54.610483 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:54.610452 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:54.756580 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:54.756105 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ptzgn"] Jul 24 17:20:54.770631 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:54.770602 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wxkxq"] Jul 24 17:20:54.772881 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:54.772854 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ca7e6f_0549_47c4_ba36_7195702ca2fd.slice/crio-f3a913bbf7356dbdabe091d7bb829aec4ee7f336ebf1e76471b0581cf7c10078 WatchSource:0}: Error finding container f3a913bbf7356dbdabe091d7bb829aec4ee7f336ebf1e76471b0581cf7c10078: Status 404 returned error can't find the container with id f3a913bbf7356dbdabe091d7bb829aec4ee7f336ebf1e76471b0581cf7c10078 Jul 24 17:20:54.774232 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:20:54.774210 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f49740_2957_426b_9a28_a1f915692dd4.slice/crio-4cc1b793d728cfa0b076181120c8d3fd1d1c21a8a42d68c816420608d1fd7ac9 WatchSource:0}: Error finding container 4cc1b793d728cfa0b076181120c8d3fd1d1c21a8a42d68c816420608d1fd7ac9: Status 404 returned error can't find the container with id 4cc1b793d728cfa0b076181120c8d3fd1d1c21a8a42d68c816420608d1fd7ac9 Jul 24 17:20:55.220935 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:55.220895 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ptzgn" event={"ID":"c7ca7e6f-0549-47c4-ba36-7195702ca2fd","Type":"ContainerStarted","Data":"f3a913bbf7356dbdabe091d7bb829aec4ee7f336ebf1e76471b0581cf7c10078"} Jul 24 17:20:55.222199 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:55.222173 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wxkxq" event={"ID":"f4f49740-2957-426b-9a28-a1f915692dd4","Type":"ContainerStarted","Data":"4cc1b793d728cfa0b076181120c8d3fd1d1c21a8a42d68c816420608d1fd7ac9"} Jul 24 17:20:58.234158 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:58.234121 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wxkxq" event={"ID":"f4f49740-2957-426b-9a28-a1f915692dd4","Type":"ContainerStarted","Data":"d74321934237f6c824da2e0b9b2119b322662bb38eb8be59e0030be3b01064d8"} Jul 24 17:20:58.234608 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:58.234180 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wxkxq" event={"ID":"f4f49740-2957-426b-9a28-a1f915692dd4","Type":"ContainerStarted","Data":"34dbe64916469005a682ee662d24b24118fcdbbcf76546e26e3ee56974589263"} Jul 24 17:20:58.234608 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:58.234253 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wxkxq" Jul 24 17:20:58.252242 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:20:58.252182 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wxkxq" podStartSLOduration=17.912940028 podStartE2EDuration="20.252160608s" totalImagesPullingTime="2.33922058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:58.251154578 +0000 UTC m=+55.903046053" watchObservedRunningTime="2026-07-24 17:20:58.252160608 +0000 UTC m=+55.904052084" Jul 24 17:21:00.243782 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:00.243701 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ptzgn" event={"ID":"c7ca7e6f-0549-47c4-ba36-7195702ca2fd","Type":"ContainerStarted","Data":"88e253d95f5fcbecca9887a7baee741792ca494795f78f7827907585a6e08cd0"} Jul 24 17:21:04.170541 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.170498 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pz4sn" Jul 24 17:21:04.257378 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.257346 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ptzgn" event={"ID":"c7ca7e6f-0549-47c4-ba36-7195702ca2fd","Type":"ContainerStarted","Data":"e689298023bc382125eec9c1c83cb9764cc90c38cb9301a13a5dc535b4632d8c"} Jul 24 17:21:04.535712 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.535600 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b987c59cf-6wpgt"] Jul 24 17:21:04.537703 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.537686 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.542540 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.542496 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 17:21:04.542666 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.542538 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mqsds\"" Jul 24 17:21:04.542666 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.542574 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 17:21:04.542666 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.542496 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 17:21:04.551675 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.551648 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 17:21:04.557972 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.557923 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b987c59cf-6wpgt"] Jul 24 17:21:04.645258 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.645219 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-t9lpl"] Jul 24 17:21:04.647397 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.647370 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk"] Jul 24 17:21:04.647588 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.647554 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-t9lpl" Jul 24 17:21:04.649160 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.649142 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl"] Jul 24 17:21:04.649275 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.649241 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" Jul 24 17:21:04.651018 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.651003 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.653130 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.653109 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 24 17:21:04.653214 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.653163 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 24 17:21:04.653214 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.653115 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 24 17:21:04.653647 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.653629 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 24 17:21:04.654027 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.653927 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 17:21:04.654027 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.653954 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-57psl\"" Jul 24 17:21:04.654027 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.653975 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 24 17:21:04.654320 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.654026 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 24 17:21:04.654491 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.654474 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 17:21:04.654615 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.654500 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 24 17:21:04.655099 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.655084 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 17:21:04.665248 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.665187 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk"] Jul 24 17:21:04.666017 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.666000 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-t9lpl"] Jul 24 17:21:04.687862 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.687823 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b987c59cf-6wpgt"] Jul 24 17:21:04.688159 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:21:04.688124 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-hrm8r registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" podUID="8101d121-c967-41a3-89ad-9568a921eded" Jul 24 17:21:04.688885 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.688859 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl"] Jul 24 17:21:04.727412 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.727364 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8101d121-c967-41a3-89ad-9568a921eded-registry-certificates\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.727608 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.727445 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrm8r\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-kube-api-access-hrm8r\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.727608 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.727582 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8101d121-c967-41a3-89ad-9568a921eded-trusted-ca\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.727689 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.727610 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-registry-tls\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.727689 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.727653 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-bound-sa-token\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.727689 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.727683 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8101d121-c967-41a3-89ad-9568a921eded-image-registry-private-configuration\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.727810 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.727715 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8101d121-c967-41a3-89ad-9568a921eded-ca-trust-extracted\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.727810 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.727761 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8101d121-c967-41a3-89ad-9568a921eded-installation-pull-secrets\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.829187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8101d121-c967-41a3-89ad-9568a921eded-installation-pull-secrets\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.829187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829139 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szfpv\" (UniqueName: \"kubernetes.io/projected/38f217c8-5f80-4e20-b304-bf1a44f7b917-kube-api-access-szfpv\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.829187 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829163 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8101d121-c967-41a3-89ad-9568a921eded-ca-trust-extracted\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.829486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829195 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8101d121-c967-41a3-89ad-9568a921eded-registry-certificates\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.829486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829355 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1b24b7a7-177c-450f-84df-9d8db31c552f-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-t9lpl\" (UID: \"1b24b7a7-177c-450f-84df-9d8db31c552f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-t9lpl" Jul 24 17:21:04.829486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829395 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrm8r\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-kube-api-access-hrm8r\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.829486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829465 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8101d121-c967-41a3-89ad-9568a921eded-trusted-ca\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.829658 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829492 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d6f46f56-bf3a-429b-89f6-a823d953f50c-klusterlet-config\") pod \"klusterlet-addon-workmgr-67db467fc-kg4xk\" (UID: \"d6f46f56-bf3a-429b-89f6-a823d953f50c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" Jul 24 17:21:04.829658 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829534 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38f217c8-5f80-4e20-b304-bf1a44f7b917-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.829658 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829559 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-registry-tls\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.829658 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829587 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-bound-sa-token\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.829658 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829616 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8101d121-c967-41a3-89ad-9568a921eded-image-registry-private-configuration\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.829658 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829644 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b24b7a7-177c-450f-84df-9d8db31c552f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-t9lpl\" (UID: \"1b24b7a7-177c-450f-84df-9d8db31c552f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-t9lpl" Jul 24 17:21:04.829955 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829669 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38f217c8-5f80-4e20-b304-bf1a44f7b917-ca\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.829955 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829693 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38f217c8-5f80-4e20-b304-bf1a44f7b917-hub\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.829955 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829709 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f217c8-5f80-4e20-b304-bf1a44f7b917-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.829955 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829717 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8101d121-c967-41a3-89ad-9568a921eded-ca-trust-extracted\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.829955 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829740 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6f46f56-bf3a-429b-89f6-a823d953f50c-tmp\") pod \"klusterlet-addon-workmgr-67db467fc-kg4xk\" (UID: \"d6f46f56-bf3a-429b-89f6-a823d953f50c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" Jul 24 17:21:04.829955 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829758 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxdlg\" (UniqueName: \"kubernetes.io/projected/d6f46f56-bf3a-429b-89f6-a823d953f50c-kube-api-access-xxdlg\") pod \"klusterlet-addon-workmgr-67db467fc-kg4xk\" (UID: \"d6f46f56-bf3a-429b-89f6-a823d953f50c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" Jul 24 17:21:04.830402 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.829986 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38f217c8-5f80-4e20-b304-bf1a44f7b917-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.830667 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.830642 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8101d121-c967-41a3-89ad-9568a921eded-registry-certificates\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.830755 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.830666 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8101d121-c967-41a3-89ad-9568a921eded-trusted-ca\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.832248 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.832223 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8101d121-c967-41a3-89ad-9568a921eded-image-registry-private-configuration\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.832370 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.832283 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8101d121-c967-41a3-89ad-9568a921eded-installation-pull-secrets\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.832505 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.832487 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-registry-tls\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.842967 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.842940 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-bound-sa-token\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.850109 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.850087 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrm8r\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-kube-api-access-hrm8r\") pod \"image-registry-7b987c59cf-6wpgt\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:04.930531 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.930482 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d6f46f56-bf3a-429b-89f6-a823d953f50c-klusterlet-config\") pod \"klusterlet-addon-workmgr-67db467fc-kg4xk\" (UID: \"d6f46f56-bf3a-429b-89f6-a823d953f50c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" Jul 24 17:21:04.930802 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.930551 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38f217c8-5f80-4e20-b304-bf1a44f7b917-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.930802 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.930591 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b24b7a7-177c-450f-84df-9d8db31c552f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-t9lpl\" (UID: \"1b24b7a7-177c-450f-84df-9d8db31c552f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-t9lpl" Jul 24 17:21:04.930802 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.930618 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38f217c8-5f80-4e20-b304-bf1a44f7b917-ca\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.930802 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.930646 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38f217c8-5f80-4e20-b304-bf1a44f7b917-hub\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.930802 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.930671 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f217c8-5f80-4e20-b304-bf1a44f7b917-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.930802 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.930701 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6f46f56-bf3a-429b-89f6-a823d953f50c-tmp\") pod \"klusterlet-addon-workmgr-67db467fc-kg4xk\" (UID: \"d6f46f56-bf3a-429b-89f6-a823d953f50c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" Jul 24 17:21:04.930802 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.930727 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxdlg\" (UniqueName: \"kubernetes.io/projected/d6f46f56-bf3a-429b-89f6-a823d953f50c-kube-api-access-xxdlg\") pod \"klusterlet-addon-workmgr-67db467fc-kg4xk\" (UID: \"d6f46f56-bf3a-429b-89f6-a823d953f50c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" Jul 24 17:21:04.930802 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.930743 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38f217c8-5f80-4e20-b304-bf1a44f7b917-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.930802 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.930804 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szfpv\" (UniqueName: \"kubernetes.io/projected/38f217c8-5f80-4e20-b304-bf1a44f7b917-kube-api-access-szfpv\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.931235 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.930829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1b24b7a7-177c-450f-84df-9d8db31c552f-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-t9lpl\" (UID: \"1b24b7a7-177c-450f-84df-9d8db31c552f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-t9lpl" Jul 24 17:21:04.931292 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.931266 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6f46f56-bf3a-429b-89f6-a823d953f50c-tmp\") pod \"klusterlet-addon-workmgr-67db467fc-kg4xk\" (UID: \"d6f46f56-bf3a-429b-89f6-a823d953f50c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" Jul 24 17:21:04.931563 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.931508 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1b24b7a7-177c-450f-84df-9d8db31c552f-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-t9lpl\" (UID: \"1b24b7a7-177c-450f-84df-9d8db31c552f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-t9lpl" Jul 24 17:21:04.931781 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.931748 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f217c8-5f80-4e20-b304-bf1a44f7b917-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.933849 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.933821 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38f217c8-5f80-4e20-b304-bf1a44f7b917-hub\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.933999 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.933976 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b24b7a7-177c-450f-84df-9d8db31c552f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-t9lpl\" (UID: \"1b24b7a7-177c-450f-84df-9d8db31c552f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-t9lpl" Jul 24 17:21:04.934784 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.934726 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d6f46f56-bf3a-429b-89f6-a823d953f50c-klusterlet-config\") pod \"klusterlet-addon-workmgr-67db467fc-kg4xk\" (UID: \"d6f46f56-bf3a-429b-89f6-a823d953f50c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" Jul 24 17:21:04.934883 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.934832 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38f217c8-5f80-4e20-b304-bf1a44f7b917-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.934936 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.934891 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38f217c8-5f80-4e20-b304-bf1a44f7b917-ca\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.935621 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.935601 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38f217c8-5f80-4e20-b304-bf1a44f7b917-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.947222 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.947192 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szfpv\" (UniqueName: \"kubernetes.io/projected/38f217c8-5f80-4e20-b304-bf1a44f7b917-kube-api-access-szfpv\") pod \"cluster-proxy-proxy-agent-84f554f89b-bnlbl\" (UID: \"38f217c8-5f80-4e20-b304-bf1a44f7b917\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:04.955153 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.955124 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxdlg\" (UniqueName: \"kubernetes.io/projected/d6f46f56-bf3a-429b-89f6-a823d953f50c-kube-api-access-xxdlg\") pod \"klusterlet-addon-workmgr-67db467fc-kg4xk\" (UID: \"d6f46f56-bf3a-429b-89f6-a823d953f50c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" Jul 24 17:21:04.957893 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.957868 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-t9lpl" Jul 24 17:21:04.963702 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.963683 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" Jul 24 17:21:04.975963 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:04.975939 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" Jul 24 17:21:05.128109 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.128031 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-t9lpl"] Jul 24 17:21:05.133920 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:21:05.133888 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b24b7a7_177c_450f_84df_9d8db31c552f.slice/crio-6933bbdfafc8dcace7cacdd2990917bc5f447cc65b4959ff0e1a0e8462980c34 WatchSource:0}: Error finding container 6933bbdfafc8dcace7cacdd2990917bc5f447cc65b4959ff0e1a0e8462980c34: Status 404 returned error can't find the container with id 6933bbdfafc8dcace7cacdd2990917bc5f447cc65b4959ff0e1a0e8462980c34 Jul 24 17:21:05.151472 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.151438 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk"] Jul 24 17:21:05.155408 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:21:05.155378 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f46f56_bf3a_429b_89f6_a823d953f50c.slice/crio-694a51a405ac9d4a9c581029b3cee5349dbe151765fd29195fa1dd38060c4a3e WatchSource:0}: Error finding container 694a51a405ac9d4a9c581029b3cee5349dbe151765fd29195fa1dd38060c4a3e: Status 404 returned error can't find the container with id 694a51a405ac9d4a9c581029b3cee5349dbe151765fd29195fa1dd38060c4a3e Jul 24 17:21:05.171396 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.171369 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl"] Jul 24 17:21:05.174492 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:21:05.174463 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38f217c8_5f80_4e20_b304_bf1a44f7b917.slice/crio-4dd4da6a961c515df74a5af0f00b68c385d25fbad9b0c255566ec3c12cb2990a WatchSource:0}: Error finding container 4dd4da6a961c515df74a5af0f00b68c385d25fbad9b0c255566ec3c12cb2990a: Status 404 returned error can't find the container with id 4dd4da6a961c515df74a5af0f00b68c385d25fbad9b0c255566ec3c12cb2990a Jul 24 17:21:05.261400 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.261363 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" event={"ID":"d6f46f56-bf3a-429b-89f6-a823d953f50c","Type":"ContainerStarted","Data":"694a51a405ac9d4a9c581029b3cee5349dbe151765fd29195fa1dd38060c4a3e"} Jul 24 17:21:05.262712 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.262681 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" event={"ID":"38f217c8-5f80-4e20-b304-bf1a44f7b917","Type":"ContainerStarted","Data":"4dd4da6a961c515df74a5af0f00b68c385d25fbad9b0c255566ec3c12cb2990a"} Jul 24 17:21:05.263969 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.263945 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:05.264094 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.263945 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-t9lpl" event={"ID":"1b24b7a7-177c-450f-84df-9d8db31c552f","Type":"ContainerStarted","Data":"6933bbdfafc8dcace7cacdd2990917bc5f447cc65b4959ff0e1a0e8462980c34"} Jul 24 17:21:05.271458 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.271431 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:05.435200 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.435108 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-registry-tls\") pod \"8101d121-c967-41a3-89ad-9568a921eded\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " Jul 24 17:21:05.435200 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.435158 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8101d121-c967-41a3-89ad-9568a921eded-image-registry-private-configuration\") pod \"8101d121-c967-41a3-89ad-9568a921eded\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " Jul 24 17:21:05.435426 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.435222 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8101d121-c967-41a3-89ad-9568a921eded-installation-pull-secrets\") pod \"8101d121-c967-41a3-89ad-9568a921eded\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " Jul 24 17:21:05.435426 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.435258 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8101d121-c967-41a3-89ad-9568a921eded-registry-certificates\") pod \"8101d121-c967-41a3-89ad-9568a921eded\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " Jul 24 17:21:05.435426 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.435286 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8101d121-c967-41a3-89ad-9568a921eded-trusted-ca\") pod \"8101d121-c967-41a3-89ad-9568a921eded\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " Jul 24 17:21:05.435426 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.435322 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrm8r\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-kube-api-access-hrm8r\") pod \"8101d121-c967-41a3-89ad-9568a921eded\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " Jul 24 17:21:05.435426 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.435348 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-bound-sa-token\") pod \"8101d121-c967-41a3-89ad-9568a921eded\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " Jul 24 17:21:05.435426 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.435380 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8101d121-c967-41a3-89ad-9568a921eded-ca-trust-extracted\") pod \"8101d121-c967-41a3-89ad-9568a921eded\" (UID: \"8101d121-c967-41a3-89ad-9568a921eded\") " Jul 24 17:21:05.436011 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.435939 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8101d121-c967-41a3-89ad-9568a921eded-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8101d121-c967-41a3-89ad-9568a921eded" (UID: "8101d121-c967-41a3-89ad-9568a921eded"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:21:05.436235 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.436186 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8101d121-c967-41a3-89ad-9568a921eded-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8101d121-c967-41a3-89ad-9568a921eded" (UID: "8101d121-c967-41a3-89ad-9568a921eded"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:21:05.436337 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.436316 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8101d121-c967-41a3-89ad-9568a921eded-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8101d121-c967-41a3-89ad-9568a921eded" (UID: "8101d121-c967-41a3-89ad-9568a921eded"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:21:05.437933 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.437898 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8101d121-c967-41a3-89ad-9568a921eded-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8101d121-c967-41a3-89ad-9568a921eded" (UID: "8101d121-c967-41a3-89ad-9568a921eded"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:21:05.438093 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.437965 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8101d121-c967-41a3-89ad-9568a921eded" (UID: "8101d121-c967-41a3-89ad-9568a921eded"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:21:05.438799 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.438775 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8101d121-c967-41a3-89ad-9568a921eded-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8101d121-c967-41a3-89ad-9568a921eded" (UID: "8101d121-c967-41a3-89ad-9568a921eded"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:21:05.438886 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.438871 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8101d121-c967-41a3-89ad-9568a921eded" (UID: "8101d121-c967-41a3-89ad-9568a921eded"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:21:05.440645 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.440622 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-kube-api-access-hrm8r" (OuterVolumeSpecName: "kube-api-access-hrm8r") pod "8101d121-c967-41a3-89ad-9568a921eded" (UID: "8101d121-c967-41a3-89ad-9568a921eded"). InnerVolumeSpecName "kube-api-access-hrm8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:21:05.536095 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.536055 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-registry-tls\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:21:05.536095 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.536099 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8101d121-c967-41a3-89ad-9568a921eded-image-registry-private-configuration\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:21:05.536339 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.536115 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8101d121-c967-41a3-89ad-9568a921eded-installation-pull-secrets\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:21:05.536339 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.536133 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8101d121-c967-41a3-89ad-9568a921eded-registry-certificates\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:21:05.536339 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.536146 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8101d121-c967-41a3-89ad-9568a921eded-trusted-ca\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:21:05.536339 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.536159 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrm8r\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-kube-api-access-hrm8r\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:21:05.536339 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.536173 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8101d121-c967-41a3-89ad-9568a921eded-bound-sa-token\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:21:05.536339 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:05.536186 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8101d121-c967-41a3-89ad-9568a921eded-ca-trust-extracted\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:21:06.267657 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:06.267624 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b987c59cf-6wpgt" Jul 24 17:21:06.312175 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:06.312139 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b987c59cf-6wpgt"] Jul 24 17:21:06.314657 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:06.314614 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b987c59cf-6wpgt"] Jul 24 17:21:06.942772 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:06.942736 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8101d121-c967-41a3-89ad-9568a921eded" path="/var/lib/kubelet/pods/8101d121-c967-41a3-89ad-9568a921eded/volumes" Jul 24 17:21:08.242945 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.242917 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wxkxq" Jul 24 17:21:08.278541 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.278473 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ptzgn" event={"ID":"c7ca7e6f-0549-47c4-ba36-7195702ca2fd","Type":"ContainerStarted","Data":"e30b7e0f21f7f99fb87bbddfda2db4e3a25aceba90192d1d3b5b743639b57248"} Jul 24 17:21:08.280307 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.280280 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-t9lpl" event={"ID":"1b24b7a7-177c-450f-84df-9d8db31c552f","Type":"ContainerStarted","Data":"adf100174c31d433aa66df74a9d210535a3852e552a85468ce2992a46c0c3d8c"} Jul 24 17:21:08.300542 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.300036 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ptzgn" podStartSLOduration=17.148959734 podStartE2EDuration="30.300013166s" totalImagesPullingTime="13.151053432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:20:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:08.298815126 +0000 UTC m=+65.950706600" watchObservedRunningTime="2026-07-24 17:21:08.300013166 +0000 UTC m=+65.951904640" Jul 24 17:21:08.315193 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.314429 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-t9lpl" podStartSLOduration=1.370710169 podStartE2EDuration="4.314406278s" totalImagesPullingTime="2.943696109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:08.313821903 +0000 UTC m=+65.965713377" watchObservedRunningTime="2026-07-24 17:21:08.314406278 +0000 UTC m=+65.966297755" Jul 24 17:21:08.562916 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.562809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:21:08.566086 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.566053 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 17:21:08.580985 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.580951 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6e8751-f60c-4f88-a369-9158d269f5ff-original-pull-secret\") pod \"global-pull-secret-syncer-7bpxv\" (UID: \"5f6e8751-f60c-4f88-a369-9158d269f5ff\") " pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:21:08.664233 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.664192 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs\") pod \"network-metrics-daemon-bzsdh\" (UID: \"505f4198-635a-4e18-847d-f85e84b9f70e\") " pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:21:08.667244 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.667219 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 17:21:08.677112 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.677086 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505f4198-635a-4e18-847d-f85e84b9f70e-metrics-certs\") pod \"network-metrics-daemon-bzsdh\" (UID: \"505f4198-635a-4e18-847d-f85e84b9f70e\") " pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:21:08.746159 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.746123 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7bpxv" Jul 24 17:21:08.764820 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.764792 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpsbc\" (UniqueName: \"kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc\") pod \"network-check-target-k42w8\" (UID: \"352eb282-95b7-442d-9834-8703f5d0db9a\") " pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:21:08.767595 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.767568 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 17:21:08.777824 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.777797 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 17:21:08.789090 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.789059 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpsbc\" (UniqueName: \"kubernetes.io/projected/352eb282-95b7-442d-9834-8703f5d0db9a-kube-api-access-jpsbc\") pod \"network-check-target-k42w8\" (UID: \"352eb282-95b7-442d-9834-8703f5d0db9a\") " pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:21:08.897285 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.897251 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7bpxv"] Jul 24 17:21:08.901005 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:21:08.900969 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6e8751_f60c_4f88_a369_9158d269f5ff.slice/crio-cb4db99d5dfb0e4de48471fec4007ee57c8f935ad14d1dcda0c50e11d10f9ecf WatchSource:0}: Error finding container cb4db99d5dfb0e4de48471fec4007ee57c8f935ad14d1dcda0c50e11d10f9ecf: Status 404 returned error can't find the container with id cb4db99d5dfb0e4de48471fec4007ee57c8f935ad14d1dcda0c50e11d10f9ecf Jul 24 17:21:08.952434 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.952403 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jwvtx\"" Jul 24 17:21:08.958104 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.958078 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8rjf6\"" Jul 24 17:21:08.960157 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.960124 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzsdh" Jul 24 17:21:08.966433 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:08.966276 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:21:09.286492 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:09.286427 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7bpxv" event={"ID":"5f6e8751-f60c-4f88-a369-9158d269f5ff","Type":"ContainerStarted","Data":"cb4db99d5dfb0e4de48471fec4007ee57c8f935ad14d1dcda0c50e11d10f9ecf"} Jul 24 17:21:09.903216 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:09.903187 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bzsdh"] Jul 24 17:21:09.907329 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:21:09.907302 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505f4198_635a_4e18_847d_f85e84b9f70e.slice/crio-5c5e28f5a47b1f6513b4f9537b9b060492e0bca200f5ef4396397a5001e6800c WatchSource:0}: Error finding container 5c5e28f5a47b1f6513b4f9537b9b060492e0bca200f5ef4396397a5001e6800c: Status 404 returned error can't find the container with id 5c5e28f5a47b1f6513b4f9537b9b060492e0bca200f5ef4396397a5001e6800c Jul 24 17:21:09.927545 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:09.924502 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k42w8"] Jul 24 17:21:10.292427 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:10.292384 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k42w8" event={"ID":"352eb282-95b7-442d-9834-8703f5d0db9a","Type":"ContainerStarted","Data":"f387bd5251ca3c435897a55ed570bd8b0b89f94d4b1ce25346d97ee49be7df6e"} Jul 24 17:21:10.293676 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:10.293642 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bzsdh" event={"ID":"505f4198-635a-4e18-847d-f85e84b9f70e","Type":"ContainerStarted","Data":"5c5e28f5a47b1f6513b4f9537b9b060492e0bca200f5ef4396397a5001e6800c"} Jul 24 17:21:12.306282 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:12.306226 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" event={"ID":"38f217c8-5f80-4e20-b304-bf1a44f7b917","Type":"ContainerStarted","Data":"f1b49e33dea8ba2c8dd174f5cf4aa9d8e87a35bf8180550edd74aa592cbc47c6"} Jul 24 17:21:12.306832 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:12.306301 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" event={"ID":"38f217c8-5f80-4e20-b304-bf1a44f7b917","Type":"ContainerStarted","Data":"3525ccaebb36b5e9f977a803f32f114e6c2d02dd63833170a5849d72ff8b3cbf"} Jul 24 17:21:12.306832 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:12.306323 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" event={"ID":"38f217c8-5f80-4e20-b304-bf1a44f7b917","Type":"ContainerStarted","Data":"79b82a303fa6bb0c47c04f264d833d0945d17d461bc775c0401236dcbf60aaa7"} Jul 24 17:21:12.331854 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:12.331527 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f554f89b-bnlbl" podStartSLOduration=2.06993297 podStartE2EDuration="8.331493569s" totalImagesPullingTime="6.261560599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:12.329747753 +0000 UTC m=+69.981639227" watchObservedRunningTime="2026-07-24 17:21:12.331493569 +0000 UTC m=+69.983385019" Jul 24 17:21:17.324836 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:17.324795 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7bpxv" event={"ID":"5f6e8751-f60c-4f88-a369-9158d269f5ff","Type":"ContainerStarted","Data":"a48bff9dc99d0c18d95dac2e08fb9cd6490a346e1041b3495accd1e9e57ce793"} Jul 24 17:21:17.326273 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:17.326242 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k42w8" event={"ID":"352eb282-95b7-442d-9834-8703f5d0db9a","Type":"ContainerStarted","Data":"a84eb7374415013995ff521b1fed5c536672bc2277cfc8373b661c694f7ac40b"} Jul 24 17:21:17.326398 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:17.326302 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:21:17.327921 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:17.327900 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bzsdh" event={"ID":"505f4198-635a-4e18-847d-f85e84b9f70e","Type":"ContainerStarted","Data":"8b3ce8de217d4d9869c672f330ef0313fcc0ae625a3ffa3344eb1c19c8868d8e"} Jul 24 17:21:17.328029 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:17.327940 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bzsdh" event={"ID":"505f4198-635a-4e18-847d-f85e84b9f70e","Type":"ContainerStarted","Data":"7eb46f638e64eba6c1647286ed2abb32a41119d95bd59fe827880d22a63fec69"} Jul 24 17:21:17.329256 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:17.329227 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" event={"ID":"d6f46f56-bf3a-429b-89f6-a823d953f50c","Type":"ContainerStarted","Data":"11f0a537c740bfe2b69ce9981d38738939ca9a23564327f057f7f46b99614ea6"} Jul 24 17:21:17.329424 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:17.329408 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" Jul 24 17:21:17.331232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:17.331213 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" Jul 24 17:21:17.342306 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:17.342263 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7bpxv" podStartSLOduration=66.979402688 podStartE2EDuration="1m14.342249888s" totalImagesPullingTime="7.3628472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:17.341403413 +0000 UTC m=+74.993294886" watchObservedRunningTime="2026-07-24 17:21:17.342249888 +0000 UTC m=+74.994141361" Jul 24 17:21:17.384352 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:17.384301 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67db467fc-kg4xk" podStartSLOduration=2.2589028620000002 podStartE2EDuration="13.384283249s" totalImagesPullingTime="11.125380387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:17.383153565 +0000 UTC m=+75.035045038" watchObservedRunningTime="2026-07-24 17:21:17.384283249 +0000 UTC m=+75.036174700" Jul 24 17:21:17.421236 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:17.421184 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k42w8" podStartSLOduration=68.068402767 podStartE2EDuration="1m14.421167593s" totalImagesPullingTime="6.352764826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:17.420653295 +0000 UTC m=+75.072544770" watchObservedRunningTime="2026-07-24 17:21:17.421167593 +0000 UTC m=+75.073059066" Jul 24 17:21:17.445311 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:17.443330 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bzsdh" podStartSLOduration=71.967120271 podStartE2EDuration="1m14.443309642s" totalImagesPullingTime="2.476189371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:17.441324055 +0000 UTC m=+75.093215530" watchObservedRunningTime="2026-07-24 17:21:17.443309642 +0000 UTC m=+75.095201117" Jul 24 17:21:28.898856 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:28.898742 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dpj5x"] Jul 24 17:21:28.945931 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:28.945893 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:28.950702 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:28.950673 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 17:21:28.950870 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:28.950693 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 17:21:28.950870 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:28.950694 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 17:21:28.950870 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:28.950731 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 17:21:28.951674 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:28.951651 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 17:21:28.951831 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:28.951662 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ggxqm\"" Jul 24 17:21:28.951831 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:28.951749 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 17:21:29.009830 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.009795 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwkxx\" (UniqueName: \"kubernetes.io/projected/6c214fce-b532-467e-9ece-94121dee447b-kube-api-access-dwkxx\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.009993 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.009844 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.009993 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.009873 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c214fce-b532-467e-9ece-94121dee447b-sys\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.009993 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.009918 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-textfile\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.009993 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.009933 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-tls\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.009993 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.009949 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c214fce-b532-467e-9ece-94121dee447b-metrics-client-ca\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.009993 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.009973 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.010203 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.010037 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-wtmp\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.010203 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.010088 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c214fce-b532-467e-9ece-94121dee447b-root\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.110395 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.110365 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-textfile\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.110395 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.110401 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-tls\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.110649 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.110418 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c214fce-b532-467e-9ece-94121dee447b-metrics-client-ca\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.110649 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.110435 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.110649 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.110531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-wtmp\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.110649 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.110587 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c214fce-b532-467e-9ece-94121dee447b-root\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.110649 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.110615 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwkxx\" (UniqueName: \"kubernetes.io/projected/6c214fce-b532-467e-9ece-94121dee447b-kube-api-access-dwkxx\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.110897 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.110674 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.110897 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.110683 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-wtmp\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.110897 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.110703 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c214fce-b532-467e-9ece-94121dee447b-root\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.110897 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.110707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c214fce-b532-467e-9ece-94121dee447b-sys\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.110897 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.110752 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c214fce-b532-467e-9ece-94121dee447b-sys\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.110897 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.110785 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-textfile\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.111189 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.111123 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c214fce-b532-467e-9ece-94121dee447b-metrics-client-ca\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.111285 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.111265 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.113016 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.112994 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-tls\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.113244 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.113225 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c214fce-b532-467e-9ece-94121dee447b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.119297 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.119269 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwkxx\" (UniqueName: \"kubernetes.io/projected/6c214fce-b532-467e-9ece-94121dee447b-kube-api-access-dwkxx\") pod \"node-exporter-dpj5x\" (UID: \"6c214fce-b532-467e-9ece-94121dee447b\") " pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.259687 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.259655 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dpj5x" Jul 24 17:21:29.292469 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:21:29.292432 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c214fce_b532_467e_9ece_94121dee447b.slice/crio-f273206b3f6d0a858d990468f840622bb5d8c33e2cb945e7b36db70a397f3dd6 WatchSource:0}: Error finding container f273206b3f6d0a858d990468f840622bb5d8c33e2cb945e7b36db70a397f3dd6: Status 404 returned error can't find the container with id f273206b3f6d0a858d990468f840622bb5d8c33e2cb945e7b36db70a397f3dd6 Jul 24 17:21:29.369571 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:29.369508 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dpj5x" event={"ID":"6c214fce-b532-467e-9ece-94121dee447b","Type":"ContainerStarted","Data":"f273206b3f6d0a858d990468f840622bb5d8c33e2cb945e7b36db70a397f3dd6"} Jul 24 17:21:31.380570 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:31.379335 2559 generic.go:356] "Generic (PLEG): container finished" podID="6c214fce-b532-467e-9ece-94121dee447b" containerID="ce9cdf155a00a8becb861dceed4927ce27bf0009baef66e095251119552759d4" exitCode=0 Jul 24 17:21:31.380570 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:31.379402 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dpj5x" event={"ID":"6c214fce-b532-467e-9ece-94121dee447b","Type":"ContainerDied","Data":"ce9cdf155a00a8becb861dceed4927ce27bf0009baef66e095251119552759d4"} Jul 24 17:21:32.384566 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:32.384531 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dpj5x" event={"ID":"6c214fce-b532-467e-9ece-94121dee447b","Type":"ContainerStarted","Data":"dd7385501980d62dc13e2878285c12910c5574240e6a88afe3a41ab7211fa9a8"} Jul 24 17:21:32.385130 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:32.384581 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dpj5x" event={"ID":"6c214fce-b532-467e-9ece-94121dee447b","Type":"ContainerStarted","Data":"61422204a15cc1a81d7361ea4ec7eb724e3dec8313c2f21d498c4b667aaa0b5b"} Jul 24 17:21:32.406605 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:32.406551 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dpj5x" podStartSLOduration=2.868500931 podStartE2EDuration="4.406533979s" totalImagesPullingTime="1.466279197s" totalInitContainerRuntime="71.753851ms" isStatefulPod=true podCreationTimestamp="2026-07-24 17:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:32.405229094 +0000 UTC m=+90.057120568" watchObservedRunningTime="2026-07-24 17:21:32.406533979 +0000 UTC m=+90.058425444" Jul 24 17:21:33.688369 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:33.688333 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4"] Jul 24 17:21:33.690559 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:33.690532 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4" Jul 24 17:21:33.692918 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:33.692897 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 24 17:21:33.693113 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:33.693097 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vk6lr\"" Jul 24 17:21:33.700722 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:33.700702 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4"] Jul 24 17:21:33.751430 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:33.751391 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktnwj\" (UniqueName: \"kubernetes.io/projected/f4d212e4-5203-4465-b795-1efa606b69c2-kube-api-access-ktnwj\") pod \"monitoring-plugin-68b7fc9574-tddr4\" (UID: \"f4d212e4-5203-4465-b795-1efa606b69c2\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4" Jul 24 17:21:33.751624 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:33.751435 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4d212e4-5203-4465-b795-1efa606b69c2-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tddr4\" (UID: \"f4d212e4-5203-4465-b795-1efa606b69c2\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4" Jul 24 17:21:33.852805 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:33.852770 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4d212e4-5203-4465-b795-1efa606b69c2-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tddr4\" (UID: \"f4d212e4-5203-4465-b795-1efa606b69c2\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4" Jul 24 17:21:33.852908 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:33.852861 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktnwj\" (UniqueName: \"kubernetes.io/projected/f4d212e4-5203-4465-b795-1efa606b69c2-kube-api-access-ktnwj\") pod \"monitoring-plugin-68b7fc9574-tddr4\" (UID: \"f4d212e4-5203-4465-b795-1efa606b69c2\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4" Jul 24 17:21:33.852945 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:21:33.852914 2559 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 24 17:21:33.852999 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:21:33.852988 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d212e4-5203-4465-b795-1efa606b69c2-monitoring-plugin-cert podName:f4d212e4-5203-4465-b795-1efa606b69c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:34.352970406 +0000 UTC m=+92.004861856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/f4d212e4-5203-4465-b795-1efa606b69c2-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-tddr4" (UID: "f4d212e4-5203-4465-b795-1efa606b69c2") : secret "monitoring-plugin-cert" not found Jul 24 17:21:33.863278 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:33.863253 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktnwj\" (UniqueName: \"kubernetes.io/projected/f4d212e4-5203-4465-b795-1efa606b69c2-kube-api-access-ktnwj\") pod \"monitoring-plugin-68b7fc9574-tddr4\" (UID: \"f4d212e4-5203-4465-b795-1efa606b69c2\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4" Jul 24 17:21:34.297483 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.297447 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9dfd84977-hqb2s"] Jul 24 17:21:34.299798 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.299782 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.302560 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.302483 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 17:21:34.302560 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.302534 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 17:21:34.302886 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.302862 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 17:21:34.302934 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.302864 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c9fdb\"" Jul 24 17:21:34.303647 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.303599 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 17:21:34.303775 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.303692 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 17:21:34.303907 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.303893 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 17:21:34.303979 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.303924 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 17:21:34.307847 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.307825 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 17:21:34.313589 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.313570 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9dfd84977-hqb2s"] Jul 24 17:21:34.355530 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.355477 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-oauth-serving-cert\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.355530 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.355534 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97b96\" (UniqueName: \"kubernetes.io/projected/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-kube-api-access-97b96\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.355733 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.355563 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4d212e4-5203-4465-b795-1efa606b69c2-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tddr4\" (UID: \"f4d212e4-5203-4465-b795-1efa606b69c2\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4" Jul 24 17:21:34.355733 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.355605 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-serving-cert\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.355733 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.355638 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-service-ca\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.355733 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.355679 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-config\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.355733 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.355706 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-oauth-config\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.355886 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.355744 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-trusted-ca-bundle\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.358093 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.358053 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4d212e4-5203-4465-b795-1efa606b69c2-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tddr4\" (UID: \"f4d212e4-5203-4465-b795-1efa606b69c2\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4" Jul 24 17:21:34.456406 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.456364 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-oauth-serving-cert\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.456622 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.456413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97b96\" (UniqueName: \"kubernetes.io/projected/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-kube-api-access-97b96\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.456622 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.456442 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-serving-cert\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.456622 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.456460 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-service-ca\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.456622 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.456486 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-config\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.456622 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.456539 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-oauth-config\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.456622 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.456568 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-trusted-ca-bundle\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.457236 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.457206 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-service-ca\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.457373 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.457353 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-config\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.457417 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.457397 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-oauth-serving-cert\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.457451 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.457433 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-trusted-ca-bundle\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.459150 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.459126 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-oauth-config\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.459262 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.459213 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-serving-cert\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.465461 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.465440 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97b96\" (UniqueName: \"kubernetes.io/projected/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-kube-api-access-97b96\") pod \"console-9dfd84977-hqb2s\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.599595 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.599483 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4" Jul 24 17:21:34.609572 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.609482 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:34.740716 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.740669 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4"] Jul 24 17:21:34.743653 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:21:34.743629 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d212e4_5203_4465_b795_1efa606b69c2.slice/crio-dd506f4704387062434512969cd9f3daee5b9938193fc0cf942b4a7a3c2a5845 WatchSource:0}: Error finding container dd506f4704387062434512969cd9f3daee5b9938193fc0cf942b4a7a3c2a5845: Status 404 returned error can't find the container with id dd506f4704387062434512969cd9f3daee5b9938193fc0cf942b4a7a3c2a5845 Jul 24 17:21:34.765457 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:34.765431 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9dfd84977-hqb2s"] Jul 24 17:21:34.768306 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:21:34.768282 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bca6b7c_4e03_4e74_856c_9e5a1800f2cc.slice/crio-db6ca0d0ade1e5203b19da376314a6ed51cc7e23dcaecb5eb4025db80784c800 WatchSource:0}: Error finding container db6ca0d0ade1e5203b19da376314a6ed51cc7e23dcaecb5eb4025db80784c800: Status 404 returned error can't find the container with id db6ca0d0ade1e5203b19da376314a6ed51cc7e23dcaecb5eb4025db80784c800 Jul 24 17:21:35.139428 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.139392 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 17:21:35.144145 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.144129 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.146644 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.146624 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 24 17:21:35.147006 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.146992 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 24 17:21:35.147475 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.147461 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 24 17:21:35.148233 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.148211 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 24 17:21:35.148326 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.148297 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e0f4r1h4e3a62\"" Jul 24 17:21:35.148575 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.148557 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 24 17:21:35.148670 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.148615 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 24 17:21:35.149486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.149462 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 24 17:21:35.149868 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.149850 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 24 17:21:35.150278 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.150257 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 24 17:21:35.150556 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.150539 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 24 17:21:35.150877 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.150857 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 24 17:21:35.151414 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.151396 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 24 17:21:35.151865 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.151845 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 24 17:21:35.156539 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.156498 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x79t8\"" Jul 24 17:21:35.158756 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.158734 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 24 17:21:35.160456 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.160277 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 24 17:21:35.163707 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.163686 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264619 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264728 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264758 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264785 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264821 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264838 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264858 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264879 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-config\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264901 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264927 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-web-config\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264948 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264967 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.264983 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55flf\" (UniqueName: \"kubernetes.io/projected/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-kube-api-access-55flf\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.265296 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.265015 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.266070 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.265030 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.266070 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.265052 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-config-out\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.266070 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.265079 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.266070 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.265099 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.365915 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.365879 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-config-out\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.365915 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.365920 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366165 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.365943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366165 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.365969 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366165 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.365998 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366165 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366020 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366165 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366058 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366165 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366095 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366165 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366127 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366165 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366164 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366580 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366190 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366580 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366580 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366236 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-config\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366580 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366253 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366580 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366282 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-web-config\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366580 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366297 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366580 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366316 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366580 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366330 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55flf\" (UniqueName: \"kubernetes.io/projected/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-kube-api-access-55flf\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366580 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366355 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.366580 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.366370 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.367683 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.367348 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.368320 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.367919 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.368320 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.368316 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.369368 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.369105 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-config-out\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.369368 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.369184 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.369555 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.369484 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.370073 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.369919 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.370221 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.370196 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.370801 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.370387 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.370801 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.370199 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.370801 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.370750 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.371337 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.371310 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.372600 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.372562 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.373088 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.373050 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-config\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.373486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.373453 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.373634 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.373501 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.373634 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.373503 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.374082 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.374056 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.374759 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.374706 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-web-config\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.379854 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.379814 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55flf\" (UniqueName: \"kubernetes.io/projected/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-kube-api-access-55flf\") pod \"prometheus-k8s-0\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.399130 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.399044 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9dfd84977-hqb2s" event={"ID":"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc","Type":"ContainerStarted","Data":"db6ca0d0ade1e5203b19da376314a6ed51cc7e23dcaecb5eb4025db80784c800"} Jul 24 17:21:35.401816 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.401780 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4" event={"ID":"f4d212e4-5203-4465-b795-1efa606b69c2","Type":"ContainerStarted","Data":"dd506f4704387062434512969cd9f3daee5b9938193fc0cf942b4a7a3c2a5845"} Jul 24 17:21:35.460089 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.460051 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:21:35.636486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:35.636449 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 17:21:35.642126 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:21:35.642095 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8357ad52_a4ae_4a93_94a9_b7f24ebe5933.slice/crio-34ed6924aea06aa01f7daa576bf5b77c643ede28c2272ce3073d3ad0b504f0c3 WatchSource:0}: Error finding container 34ed6924aea06aa01f7daa576bf5b77c643ede28c2272ce3073d3ad0b504f0c3: Status 404 returned error can't find the container with id 34ed6924aea06aa01f7daa576bf5b77c643ede28c2272ce3073d3ad0b504f0c3 Jul 24 17:21:36.406958 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:36.406919 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerStarted","Data":"34ed6924aea06aa01f7daa576bf5b77c643ede28c2272ce3073d3ad0b504f0c3"} Jul 24 17:21:37.412471 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:37.412425 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4" event={"ID":"f4d212e4-5203-4465-b795-1efa606b69c2","Type":"ContainerStarted","Data":"ec1890cce2b59a4c8f938a34030a24bbb640e7d94be55108790a1f1693efcf00"} Jul 24 17:21:37.413058 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:37.413037 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4" Jul 24 17:21:37.418615 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:37.418570 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4" Jul 24 17:21:37.431596 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:37.431536 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tddr4" podStartSLOduration=2.637767989 podStartE2EDuration="4.431496795s" totalImagesPullingTime="1.793728806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:37.429774459 +0000 UTC m=+95.081665944" watchObservedRunningTime="2026-07-24 17:21:37.431496795 +0000 UTC m=+95.083388267" Jul 24 17:21:39.420769 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:39.420729 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9dfd84977-hqb2s" event={"ID":"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc","Type":"ContainerStarted","Data":"162d12ec8132919c59332307954a6bd01a6f130f60b72f977199e6dec3477f65"} Jul 24 17:21:39.422421 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:39.422395 2559 generic.go:356] "Generic (PLEG): container finished" podID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerID="db07be900b7a3334e843b369cd824aba11c725ad5346e51afe6fcfc254910e0c" exitCode=0 Jul 24 17:21:39.422565 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:39.422474 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerDied","Data":"db07be900b7a3334e843b369cd824aba11c725ad5346e51afe6fcfc254910e0c"} Jul 24 17:21:39.439109 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:39.439059 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9dfd84977-hqb2s" podStartSLOduration=1.7711244339999999 podStartE2EDuration="5.439043894s" totalImagesPullingTime="3.66791946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:39.437951437 +0000 UTC m=+97.089842910" watchObservedRunningTime="2026-07-24 17:21:39.439043894 +0000 UTC m=+97.090935367" Jul 24 17:21:40.199083 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:40.199044 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9dfd84977-hqb2s"] Jul 24 17:21:44.445793 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:44.445738 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerStarted","Data":"946016a81fd6221df79732460a5b301b48675c3f2495ada7135028a54a773342"} Jul 24 17:21:44.446258 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:44.445839 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerStarted","Data":"107ed1aa0eafb82e88d5cb740ae26a5cbddd92811262b94ec8f3834fdc0f489a"} Jul 24 17:21:44.609906 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:44.609873 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:21:46.459386 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:46.459293 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerStarted","Data":"335460ddde628c95db3f5107f3ac496d3612a78148fc3329a43a32f79ca4b16a"} Jul 24 17:21:46.459859 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:46.459405 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerStarted","Data":"84eb5b036383d755aba0fb69ae13a43ced7c19530113a71ad02362b229613608"} Jul 24 17:21:46.459859 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:46.459455 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerStarted","Data":"611ad87e7f75a62fac8c7d13ab82d74746127f23ee3a4a407398afc1aed6edce"} Jul 24 17:21:46.459859 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:46.459494 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerStarted","Data":"ad551841978fb2601be7e29d2f31fe633dfd71b9a0f510b24ceda0a6aff246d3"} Jul 24 17:21:46.489607 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:46.489550 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.493256658 podStartE2EDuration="11.489533947s" totalImagesPullingTime="8.979727788s" totalInitContainerRuntime="16.549501ms" isStatefulPod=false podCreationTimestamp="2026-07-24 17:21:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:46.488151785 +0000 UTC m=+104.140043257" watchObservedRunningTime="2026-07-24 17:21:46.489533947 +0000 UTC m=+104.141425410" Jul 24 17:21:48.335218 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:48.335184 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k42w8" Jul 24 17:21:50.460262 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:21:50.460218 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:06.454239 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.454148 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9dfd84977-hqb2s" podUID="1bca6b7c-4e03-4e74-856c-9e5a1800f2cc" containerName="console" containerID="cri-o://162d12ec8132919c59332307954a6bd01a6f130f60b72f977199e6dec3477f65" gracePeriod=15 Jul 24 17:22:06.712844 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.712774 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:22:06.732940 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.732913 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-service-ca\") pod \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " Jul 24 17:22:06.733119 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.732967 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97b96\" (UniqueName: \"kubernetes.io/projected/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-kube-api-access-97b96\") pod \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " Jul 24 17:22:06.733119 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.733002 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-config\") pod \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " Jul 24 17:22:06.733119 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.733041 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-oauth-serving-cert\") pod \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " Jul 24 17:22:06.733119 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.733078 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-serving-cert\") pod \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " Jul 24 17:22:06.733119 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.733098 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-trusted-ca-bundle\") pod \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " Jul 24 17:22:06.733391 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.733155 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-oauth-config\") pod \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\" (UID: \"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc\") " Jul 24 17:22:06.733860 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.733808 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-service-ca" (OuterVolumeSpecName: "service-ca") pod "1bca6b7c-4e03-4e74-856c-9e5a1800f2cc" (UID: "1bca6b7c-4e03-4e74-856c-9e5a1800f2cc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:22:06.734081 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.733896 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1bca6b7c-4e03-4e74-856c-9e5a1800f2cc" (UID: "1bca6b7c-4e03-4e74-856c-9e5a1800f2cc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:22:06.734195 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.734169 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-config" (OuterVolumeSpecName: "console-config") pod "1bca6b7c-4e03-4e74-856c-9e5a1800f2cc" (UID: "1bca6b7c-4e03-4e74-856c-9e5a1800f2cc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:22:06.734297 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.734273 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1bca6b7c-4e03-4e74-856c-9e5a1800f2cc" (UID: "1bca6b7c-4e03-4e74-856c-9e5a1800f2cc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:22:06.736225 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.736076 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1bca6b7c-4e03-4e74-856c-9e5a1800f2cc" (UID: "1bca6b7c-4e03-4e74-856c-9e5a1800f2cc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:22:06.739102 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.736666 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1bca6b7c-4e03-4e74-856c-9e5a1800f2cc" (UID: "1bca6b7c-4e03-4e74-856c-9e5a1800f2cc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:22:06.739102 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.736742 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-kube-api-access-97b96" (OuterVolumeSpecName: "kube-api-access-97b96") pod "1bca6b7c-4e03-4e74-856c-9e5a1800f2cc" (UID: "1bca6b7c-4e03-4e74-856c-9e5a1800f2cc"). InnerVolumeSpecName "kube-api-access-97b96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:22:06.834036 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.833899 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-oauth-config\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:06.834036 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.833938 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-service-ca\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:06.834036 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.833954 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97b96\" (UniqueName: \"kubernetes.io/projected/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-kube-api-access-97b96\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:06.834036 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.833968 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-config\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:06.834036 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.833982 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-oauth-serving-cert\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:06.834036 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.833995 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-console-serving-cert\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:06.834036 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:06.834008 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc-trusted-ca-bundle\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:07.540098 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:07.540063 2559 generic.go:356] "Generic (PLEG): container finished" podID="1bca6b7c-4e03-4e74-856c-9e5a1800f2cc" containerID="162d12ec8132919c59332307954a6bd01a6f130f60b72f977199e6dec3477f65" exitCode=0 Jul 24 17:22:07.540486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:07.540116 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9dfd84977-hqb2s" Jul 24 17:22:07.540486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:07.540135 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9dfd84977-hqb2s" event={"ID":"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc","Type":"ContainerDied","Data":"162d12ec8132919c59332307954a6bd01a6f130f60b72f977199e6dec3477f65"} Jul 24 17:22:07.540486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:07.540187 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9dfd84977-hqb2s" event={"ID":"1bca6b7c-4e03-4e74-856c-9e5a1800f2cc","Type":"ContainerDied","Data":"db6ca0d0ade1e5203b19da376314a6ed51cc7e23dcaecb5eb4025db80784c800"} Jul 24 17:22:07.540486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:07.540213 2559 scope.go:117] "RemoveContainer" containerID="162d12ec8132919c59332307954a6bd01a6f130f60b72f977199e6dec3477f65" Jul 24 17:22:07.549342 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:07.549318 2559 scope.go:117] "RemoveContainer" containerID="162d12ec8132919c59332307954a6bd01a6f130f60b72f977199e6dec3477f65" Jul 24 17:22:07.549693 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:22:07.549663 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162d12ec8132919c59332307954a6bd01a6f130f60b72f977199e6dec3477f65\": container with ID starting with 162d12ec8132919c59332307954a6bd01a6f130f60b72f977199e6dec3477f65 not found: ID does not exist" containerID="162d12ec8132919c59332307954a6bd01a6f130f60b72f977199e6dec3477f65" Jul 24 17:22:07.549774 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:07.549704 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162d12ec8132919c59332307954a6bd01a6f130f60b72f977199e6dec3477f65"} err="failed to get container status \"162d12ec8132919c59332307954a6bd01a6f130f60b72f977199e6dec3477f65\": rpc error: code = NotFound desc = could not find container \"162d12ec8132919c59332307954a6bd01a6f130f60b72f977199e6dec3477f65\": container with ID starting with 162d12ec8132919c59332307954a6bd01a6f130f60b72f977199e6dec3477f65 not found: ID does not exist" Jul 24 17:22:07.557591 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:07.557557 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9dfd84977-hqb2s"] Jul 24 17:22:07.563651 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:07.563587 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9dfd84977-hqb2s"] Jul 24 17:22:08.941324 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:08.941286 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bca6b7c-4e03-4e74-856c-9e5a1800f2cc" path="/var/lib/kubelet/pods/1bca6b7c-4e03-4e74-856c-9e5a1800f2cc/volumes" Jul 24 17:22:35.461063 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:35.461026 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:35.480639 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:35.480614 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:35.653711 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:35.653686 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:53.500086 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.500013 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 17:22:53.500539 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.500348 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="kube-rbac-proxy" containerID="cri-o://84eb5b036383d755aba0fb69ae13a43ced7c19530113a71ad02362b229613608" gracePeriod=600 Jul 24 17:22:53.500539 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.500392 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="kube-rbac-proxy-thanos" containerID="cri-o://335460ddde628c95db3f5107f3ac496d3612a78148fc3329a43a32f79ca4b16a" gracePeriod=600 Jul 24 17:22:53.500539 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.500394 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="thanos-sidecar" containerID="cri-o://ad551841978fb2601be7e29d2f31fe633dfd71b9a0f510b24ceda0a6aff246d3" gracePeriod=600 Jul 24 17:22:53.500539 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.500422 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="config-reloader" containerID="cri-o://946016a81fd6221df79732460a5b301b48675c3f2495ada7135028a54a773342" gracePeriod=600 Jul 24 17:22:53.500539 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.500491 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="prometheus" containerID="cri-o://107ed1aa0eafb82e88d5cb740ae26a5cbddd92811262b94ec8f3834fdc0f489a" gracePeriod=600 Jul 24 17:22:53.500539 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.500431 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="kube-rbac-proxy-web" containerID="cri-o://611ad87e7f75a62fac8c7d13ab82d74746127f23ee3a4a407398afc1aed6edce" gracePeriod=600 Jul 24 17:22:53.703109 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.703078 2559 generic.go:356] "Generic (PLEG): container finished" podID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerID="335460ddde628c95db3f5107f3ac496d3612a78148fc3329a43a32f79ca4b16a" exitCode=0 Jul 24 17:22:53.703109 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.703102 2559 generic.go:356] "Generic (PLEG): container finished" podID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerID="84eb5b036383d755aba0fb69ae13a43ced7c19530113a71ad02362b229613608" exitCode=0 Jul 24 17:22:53.703109 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.703109 2559 generic.go:356] "Generic (PLEG): container finished" podID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerID="611ad87e7f75a62fac8c7d13ab82d74746127f23ee3a4a407398afc1aed6edce" exitCode=0 Jul 24 17:22:53.703109 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.703115 2559 generic.go:356] "Generic (PLEG): container finished" podID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerID="ad551841978fb2601be7e29d2f31fe633dfd71b9a0f510b24ceda0a6aff246d3" exitCode=0 Jul 24 17:22:53.703109 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.703120 2559 generic.go:356] "Generic (PLEG): container finished" podID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerID="946016a81fd6221df79732460a5b301b48675c3f2495ada7135028a54a773342" exitCode=0 Jul 24 17:22:53.703109 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.703125 2559 generic.go:356] "Generic (PLEG): container finished" podID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerID="107ed1aa0eafb82e88d5cb740ae26a5cbddd92811262b94ec8f3834fdc0f489a" exitCode=0 Jul 24 17:22:53.703435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.703153 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerDied","Data":"335460ddde628c95db3f5107f3ac496d3612a78148fc3329a43a32f79ca4b16a"} Jul 24 17:22:53.703435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.703216 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerDied","Data":"84eb5b036383d755aba0fb69ae13a43ced7c19530113a71ad02362b229613608"} Jul 24 17:22:53.703435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.703248 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerDied","Data":"611ad87e7f75a62fac8c7d13ab82d74746127f23ee3a4a407398afc1aed6edce"} Jul 24 17:22:53.703435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.703273 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerDied","Data":"ad551841978fb2601be7e29d2f31fe633dfd71b9a0f510b24ceda0a6aff246d3"} Jul 24 17:22:53.703435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.703295 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerDied","Data":"946016a81fd6221df79732460a5b301b48675c3f2495ada7135028a54a773342"} Jul 24 17:22:53.703435 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.703319 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerDied","Data":"107ed1aa0eafb82e88d5cb740ae26a5cbddd92811262b94ec8f3834fdc0f489a"} Jul 24 17:22:53.743176 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.743153 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:53.903353 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903285 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-config-out\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.903353 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903344 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55flf\" (UniqueName: \"kubernetes.io/projected/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-kube-api-access-55flf\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.903353 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903367 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-trusted-ca-bundle\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.903703 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903394 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-2\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.903703 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903413 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-thanos-prometheus-http-client-file\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.903703 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903444 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-web-config\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.903703 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903462 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-metrics-client-certs\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.903703 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903480 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-metrics-client-ca\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.903703 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903501 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-1\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.903703 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903551 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-kubelet-serving-ca-bundle\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.903703 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903580 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-tls\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.903703 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903613 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-config\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.903703 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903655 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.903703 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903683 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-serving-certs-ca-bundle\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.904214 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903712 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.904214 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903737 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-kube-rbac-proxy\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.904214 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903768 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-grpc-tls\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.904214 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903781 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:22:53.904214 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903801 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-0\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.904214 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.903823 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-tls-assets\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.904693 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.904274 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:22:53.904693 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.904401 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-db\") pod \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\" (UID: \"8357ad52-a4ae-4a93-94a9-b7f24ebe5933\") " Jul 24 17:22:53.904693 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.904619 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:22:53.904693 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.904632 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:22:53.904693 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.904664 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:53.904693 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.904683 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:53.905003 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.904870 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:22:53.907346 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.905123 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:22:53.907346 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.907016 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-config-out" (OuterVolumeSpecName: "config-out") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:22:53.907918 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.907595 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:22:53.907918 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.907898 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:22:53.907918 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.907896 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:22:53.908134 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.907977 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:22:53.908292 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.908259 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-kube-api-access-55flf" (OuterVolumeSpecName: "kube-api-access-55flf") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "kube-api-access-55flf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:22:53.908368 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.908334 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:22:53.908567 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.908541 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:22:53.908893 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.908870 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:22:53.909017 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.909001 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:22:53.909195 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.909177 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-config" (OuterVolumeSpecName: "config") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:22:53.909280 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.909259 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:22:53.909426 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.909404 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:22:53.917266 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:53.917241 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-web-config" (OuterVolumeSpecName: "web-config") pod "8357ad52-a4ae-4a93-94a9-b7f24ebe5933" (UID: "8357ad52-a4ae-4a93-94a9-b7f24ebe5933"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:22:54.005931 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.005895 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.005931 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.005925 2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-tls-assets\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.005931 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.005937 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-db\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.005946 2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-config-out\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.005955 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55flf\" (UniqueName: \"kubernetes.io/projected/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-kube-api-access-55flf\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.005964 2559 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.005972 2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-web-config\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.005980 2559 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-metrics-client-certs\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.005987 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-metrics-client-ca\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.005996 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.006006 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.006014 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.006021 2559 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-config\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.006029 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.006037 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.006045 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.006054 2559 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-kube-rbac-proxy\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.006163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.006062 2559 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8357ad52-a4ae-4a93-94a9-b7f24ebe5933-secret-grpc-tls\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:22:54.712159 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.712126 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8357ad52-a4ae-4a93-94a9-b7f24ebe5933","Type":"ContainerDied","Data":"34ed6924aea06aa01f7daa576bf5b77c643ede28c2272ce3073d3ad0b504f0c3"} Jul 24 17:22:54.712595 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.712188 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.712595 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.712203 2559 scope.go:117] "RemoveContainer" containerID="335460ddde628c95db3f5107f3ac496d3612a78148fc3329a43a32f79ca4b16a" Jul 24 17:22:54.719856 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.719832 2559 scope.go:117] "RemoveContainer" containerID="84eb5b036383d755aba0fb69ae13a43ced7c19530113a71ad02362b229613608" Jul 24 17:22:54.729030 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.729011 2559 scope.go:117] "RemoveContainer" containerID="611ad87e7f75a62fac8c7d13ab82d74746127f23ee3a4a407398afc1aed6edce" Jul 24 17:22:54.735824 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.735804 2559 scope.go:117] "RemoveContainer" containerID="ad551841978fb2601be7e29d2f31fe633dfd71b9a0f510b24ceda0a6aff246d3" Jul 24 17:22:54.737365 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.737346 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 17:22:54.741765 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.741741 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 17:22:54.743096 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.743081 2559 scope.go:117] "RemoveContainer" containerID="946016a81fd6221df79732460a5b301b48675c3f2495ada7135028a54a773342" Jul 24 17:22:54.749565 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.749547 2559 scope.go:117] "RemoveContainer" containerID="107ed1aa0eafb82e88d5cb740ae26a5cbddd92811262b94ec8f3834fdc0f489a" Jul 24 17:22:54.756336 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.756322 2559 scope.go:117] "RemoveContainer" containerID="db07be900b7a3334e843b369cd824aba11c725ad5346e51afe6fcfc254910e0c" Jul 24 17:22:54.771792 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.771736 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 17:22:54.772026 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772014 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="init-config-reloader" Jul 24 17:22:54.772062 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772029 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="init-config-reloader" Jul 24 17:22:54.772062 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772038 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bca6b7c-4e03-4e74-856c-9e5a1800f2cc" containerName="console" Jul 24 17:22:54.772062 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772044 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bca6b7c-4e03-4e74-856c-9e5a1800f2cc" containerName="console" Jul 24 17:22:54.772062 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772051 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="config-reloader" Jul 24 17:22:54.772062 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772056 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="config-reloader" Jul 24 17:22:54.772062 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772062 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="kube-rbac-proxy-web" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772069 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="kube-rbac-proxy-web" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772078 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="kube-rbac-proxy" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772085 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="kube-rbac-proxy" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772093 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="kube-rbac-proxy-thanos" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772099 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="kube-rbac-proxy-thanos" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772112 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="thanos-sidecar" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772116 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="thanos-sidecar" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772123 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="prometheus" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772127 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="prometheus" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772168 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bca6b7c-4e03-4e74-856c-9e5a1800f2cc" containerName="console" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772176 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="thanos-sidecar" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772182 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="kube-rbac-proxy-web" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772187 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="kube-rbac-proxy-thanos" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772193 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="prometheus" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772198 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="kube-rbac-proxy" Jul 24 17:22:54.772232 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.772204 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" containerName="config-reloader" Jul 24 17:22:54.775856 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.775840 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.785336 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.785309 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 24 17:22:54.785446 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.785386 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 24 17:22:54.785446 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.785322 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 24 17:22:54.785624 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.785597 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 24 17:22:54.785679 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.785601 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 24 17:22:54.785797 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.785781 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 24 17:22:54.785857 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.785799 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 24 17:22:54.785857 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.785787 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 24 17:22:54.785961 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.785882 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e0f4r1h4e3a62\"" Jul 24 17:22:54.785961 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.785891 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 24 17:22:54.786208 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.786192 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 24 17:22:54.786276 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.786209 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 24 17:22:54.786276 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.786216 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x79t8\"" Jul 24 17:22:54.786276 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.786262 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 24 17:22:54.786649 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.786635 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 24 17:22:54.787874 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.787859 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 24 17:22:54.793736 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.793716 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 24 17:22:54.795620 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.795602 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 17:22:54.911855 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.911814 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.911855 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.911861 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912096 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.911880 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2wg9\" (UniqueName: \"kubernetes.io/projected/6e3734a0-d1b3-4f6a-972c-91257d67d902-kube-api-access-w2wg9\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912096 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.911903 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912096 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.911930 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912096 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.911950 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912096 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.911965 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912096 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.911994 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912096 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.912020 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-config\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912096 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.912036 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912096 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.912069 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e3734a0-d1b3-4f6a-972c-91257d67d902-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912096 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.912097 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e3734a0-d1b3-4f6a-972c-91257d67d902-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912425 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.912117 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912425 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.912135 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912425 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.912217 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912425 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.912237 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912425 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.912255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912425 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.912279 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912425 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.912299 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.912425 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.912314 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:54.941465 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:54.941416 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8357ad52-a4ae-4a93-94a9-b7f24ebe5933" path="/var/lib/kubelet/pods/8357ad52-a4ae-4a93-94a9-b7f24ebe5933/volumes" Jul 24 17:22:55.013155 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013108 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013155 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013156 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2wg9\" (UniqueName: \"kubernetes.io/projected/6e3734a0-d1b3-4f6a-972c-91257d67d902-kube-api-access-w2wg9\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013407 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013407 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013214 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013407 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013246 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013407 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013270 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013407 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013298 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013407 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013324 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-config\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013407 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013345 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013407 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013372 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e3734a0-d1b3-4f6a-972c-91257d67d902-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013407 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013401 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e3734a0-d1b3-4f6a-972c-91257d67d902-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013877 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013877 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013447 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013877 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013484 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013877 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013509 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013877 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013564 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013877 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013593 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013877 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013627 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013877 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013652 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.013877 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013679 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.014483 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013914 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.014483 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013962 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.014483 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.013983 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.014483 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.014278 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.015223 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.014867 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.015223 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.015152 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.016808 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.016621 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.016808 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.016651 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.016808 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.016749 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.016808 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.016785 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e3734a0-d1b3-4f6a-972c-91257d67d902-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.017074 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.016884 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e3734a0-d1b3-4f6a-972c-91257d67d902-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.017132 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.017114 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.017708 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.017667 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.017876 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.017856 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-config\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.019298 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.019270 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.019378 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.019361 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e3734a0-d1b3-4f6a-972c-91257d67d902-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.019884 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.019858 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.019958 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.019903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.020238 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.020217 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e3734a0-d1b3-4f6a-972c-91257d67d902-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.023146 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.023098 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2wg9\" (UniqueName: \"kubernetes.io/projected/6e3734a0-d1b3-4f6a-972c-91257d67d902-kube-api-access-w2wg9\") pod \"prometheus-k8s-0\" (UID: \"6e3734a0-d1b3-4f6a-972c-91257d67d902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.084778 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.084748 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:22:55.225202 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.225164 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 17:22:55.228068 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:22:55.228034 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3734a0_d1b3_4f6a_972c_91257d67d902.slice/crio-db2c8db0ec329108cad12e35edeafb4d7e185db5ba1b58daa0c7f2410a5dc184 WatchSource:0}: Error finding container db2c8db0ec329108cad12e35edeafb4d7e185db5ba1b58daa0c7f2410a5dc184: Status 404 returned error can't find the container with id db2c8db0ec329108cad12e35edeafb4d7e185db5ba1b58daa0c7f2410a5dc184 Jul 24 17:22:55.716758 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.716673 2559 generic.go:356] "Generic (PLEG): container finished" podID="6e3734a0-d1b3-4f6a-972c-91257d67d902" containerID="0f0c74e7dfd3885cdee6f480d15a41d6703dc2ab687c535261287203540b05d4" exitCode=0 Jul 24 17:22:55.717127 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.716757 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e3734a0-d1b3-4f6a-972c-91257d67d902","Type":"ContainerDied","Data":"0f0c74e7dfd3885cdee6f480d15a41d6703dc2ab687c535261287203540b05d4"} Jul 24 17:22:55.717127 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:55.716837 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e3734a0-d1b3-4f6a-972c-91257d67d902","Type":"ContainerStarted","Data":"db2c8db0ec329108cad12e35edeafb4d7e185db5ba1b58daa0c7f2410a5dc184"} Jul 24 17:22:56.724593 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:56.724559 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e3734a0-d1b3-4f6a-972c-91257d67d902","Type":"ContainerStarted","Data":"bfdb0880309d1b201a8514510a05b1cbaaef211619cd7e9df066bc4c69fcf231"} Jul 24 17:22:56.724939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:56.724653 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e3734a0-d1b3-4f6a-972c-91257d67d902","Type":"ContainerStarted","Data":"2a005b5ec448816e3e05679ac7a8d69aa3095fa39b3f20d90fb224e3f08918a8"} Jul 24 17:22:56.724939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:56.724678 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e3734a0-d1b3-4f6a-972c-91257d67d902","Type":"ContainerStarted","Data":"1d37ec7629ab7055f46388c7bfdd6481e1c6c489b0e335c72626f3f8d66c376c"} Jul 24 17:22:56.724939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:56.724706 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e3734a0-d1b3-4f6a-972c-91257d67d902","Type":"ContainerStarted","Data":"529c9d6dde95aac3c6af3a71e567fd76fecdfafeeef334444bff2ed0325e7f9a"} Jul 24 17:22:56.724939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:56.724728 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e3734a0-d1b3-4f6a-972c-91257d67d902","Type":"ContainerStarted","Data":"8b0622aa9fae80943276f9d8f178f3773632b763f71b45d314f4c62bf57d55c3"} Jul 24 17:22:56.724939 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:22:56.724751 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e3734a0-d1b3-4f6a-972c-91257d67d902","Type":"ContainerStarted","Data":"273c57ab963d9781cad91e29dd48fe7aa622f4e554a187eda2d37f6fe4dc169a"} Jul 24 17:23:00.085214 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:23:00.085174 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:23:55.085685 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:23:55.085646 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:23:55.101301 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:23:55.101277 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:23:55.130493 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:23:55.130442 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=61.11304437 podStartE2EDuration="1m1.130427718s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.383348ms" isStatefulPod=false podCreationTimestamp="2026-07-24 17:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:56.75793508 +0000 UTC m=+174.409826553" watchObservedRunningTime="2026-07-24 17:23:55.130427718 +0000 UTC m=+232.782319192" Jul 24 17:23:55.941105 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:23:55.941071 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 17:25:02.766594 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:02.766570 2559 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 17:25:20.384183 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.384149 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k"] Jul 24 17:25:20.387359 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.387343 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:20.391270 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.391223 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vrv2x\"" Jul 24 17:25:20.391270 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.391238 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 17:25:20.391270 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.391236 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 24 17:25:20.391270 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.391267 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 24 17:25:20.391612 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.391329 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 24 17:25:20.391612 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.391379 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:25:20.399798 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.399772 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k"] Jul 24 17:25:20.502831 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.502803 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca87879c-f84d-42b9-874b-5fe5e3fa3faf-metrics-cert\") pod \"lws-controller-manager-bdcf88f84-zx57k\" (UID: \"ca87879c-f84d-42b9-874b-5fe5e3fa3faf\") " pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:20.502831 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.502830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jrk4\" (UniqueName: \"kubernetes.io/projected/ca87879c-f84d-42b9-874b-5fe5e3fa3faf-kube-api-access-7jrk4\") pod \"lws-controller-manager-bdcf88f84-zx57k\" (UID: \"ca87879c-f84d-42b9-874b-5fe5e3fa3faf\") " pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:20.503011 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.502851 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca87879c-f84d-42b9-874b-5fe5e3fa3faf-cert\") pod \"lws-controller-manager-bdcf88f84-zx57k\" (UID: \"ca87879c-f84d-42b9-874b-5fe5e3fa3faf\") " pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:20.503011 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.502936 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ca87879c-f84d-42b9-874b-5fe5e3fa3faf-manager-config\") pod \"lws-controller-manager-bdcf88f84-zx57k\" (UID: \"ca87879c-f84d-42b9-874b-5fe5e3fa3faf\") " pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:20.603540 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.603484 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca87879c-f84d-42b9-874b-5fe5e3fa3faf-metrics-cert\") pod \"lws-controller-manager-bdcf88f84-zx57k\" (UID: \"ca87879c-f84d-42b9-874b-5fe5e3fa3faf\") " pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:20.603540 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.603541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jrk4\" (UniqueName: \"kubernetes.io/projected/ca87879c-f84d-42b9-874b-5fe5e3fa3faf-kube-api-access-7jrk4\") pod \"lws-controller-manager-bdcf88f84-zx57k\" (UID: \"ca87879c-f84d-42b9-874b-5fe5e3fa3faf\") " pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:20.603765 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.603569 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca87879c-f84d-42b9-874b-5fe5e3fa3faf-cert\") pod \"lws-controller-manager-bdcf88f84-zx57k\" (UID: \"ca87879c-f84d-42b9-874b-5fe5e3fa3faf\") " pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:20.603765 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.603623 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ca87879c-f84d-42b9-874b-5fe5e3fa3faf-manager-config\") pod \"lws-controller-manager-bdcf88f84-zx57k\" (UID: \"ca87879c-f84d-42b9-874b-5fe5e3fa3faf\") " pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:20.604230 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.604207 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ca87879c-f84d-42b9-874b-5fe5e3fa3faf-manager-config\") pod \"lws-controller-manager-bdcf88f84-zx57k\" (UID: \"ca87879c-f84d-42b9-874b-5fe5e3fa3faf\") " pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:20.606164 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.606147 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca87879c-f84d-42b9-874b-5fe5e3fa3faf-cert\") pod \"lws-controller-manager-bdcf88f84-zx57k\" (UID: \"ca87879c-f84d-42b9-874b-5fe5e3fa3faf\") " pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:20.606224 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.606206 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca87879c-f84d-42b9-874b-5fe5e3fa3faf-metrics-cert\") pod \"lws-controller-manager-bdcf88f84-zx57k\" (UID: \"ca87879c-f84d-42b9-874b-5fe5e3fa3faf\") " pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:20.613671 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.613651 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jrk4\" (UniqueName: \"kubernetes.io/projected/ca87879c-f84d-42b9-874b-5fe5e3fa3faf-kube-api-access-7jrk4\") pod \"lws-controller-manager-bdcf88f84-zx57k\" (UID: \"ca87879c-f84d-42b9-874b-5fe5e3fa3faf\") " pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:20.697041 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.696970 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:20.823707 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.823682 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k"] Jul 24 17:25:20.825971 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:25:20.825948 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca87879c_f84d_42b9_874b_5fe5e3fa3faf.slice/crio-6f79399c2bdee83762bc5130e509fddb82417abc4062080016c1ea26cb9bea36 WatchSource:0}: Error finding container 6f79399c2bdee83762bc5130e509fddb82417abc4062080016c1ea26cb9bea36: Status 404 returned error can't find the container with id 6f79399c2bdee83762bc5130e509fddb82417abc4062080016c1ea26cb9bea36 Jul 24 17:25:20.827803 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:20.827786 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:25:21.211394 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:21.211359 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" event={"ID":"ca87879c-f84d-42b9-874b-5fe5e3fa3faf","Type":"ContainerStarted","Data":"6f79399c2bdee83762bc5130e509fddb82417abc4062080016c1ea26cb9bea36"} Jul 24 17:25:24.224279 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.224240 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" event={"ID":"ca87879c-f84d-42b9-874b-5fe5e3fa3faf","Type":"ContainerStarted","Data":"653daefbd8e10ea6feccbea10fad97a0e6299c70e0d0e22a636c714db95006e8"} Jul 24 17:25:24.224702 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.224391 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:24.242362 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.242320 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" podStartSLOduration=1.057585891 podStartE2EDuration="4.242306752s" totalImagesPullingTime="3.184720861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:25:24.241579613 +0000 UTC m=+321.893471087" watchObservedRunningTime="2026-07-24 17:25:24.242306752 +0000 UTC m=+321.894198225" Jul 24 17:25:24.552982 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.552946 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb"] Jul 24 17:25:24.556316 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.556299 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.561974 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.561942 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 17:25:24.562425 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.562404 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2dnfg\"" Jul 24 17:25:24.563055 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.563041 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 17:25:24.563204 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.563192 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 17:25:24.563762 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.563744 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 17:25:24.581474 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.581447 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb"] Jul 24 17:25:24.637710 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.637681 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.637710 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.637715 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.637943 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.637734 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nchsd\" (UniqueName: \"kubernetes.io/projected/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-kube-api-access-nchsd\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.637943 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.637855 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-manifests\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.637943 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.637897 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-tmp\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.738833 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.738800 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.738833 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.738832 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.739000 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.738853 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nchsd\" (UniqueName: \"kubernetes.io/projected/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-kube-api-access-nchsd\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.739000 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.738898 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-manifests\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.739000 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.738922 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-tmp\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.739342 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.739320 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-manifests\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.741309 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.741290 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-tmp\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.741419 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.741403 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.741494 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.741477 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.748902 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.748876 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nchsd\" (UniqueName: \"kubernetes.io/projected/4b19c3e1-fd2f-4538-acf1-8013eec8ed9b-kube-api-access-nchsd\") pod \"opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb\" (UID: \"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b\") " pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:24.865540 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:24.865414 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:25.007551 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:25.007503 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb"] Jul 24 17:25:25.009499 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:25:25.009478 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b19c3e1_fd2f_4538_acf1_8013eec8ed9b.slice/crio-a8c585206c85c520a8c37a04845be0790ce5e61bbc3430a7f14f1943077cb96b WatchSource:0}: Error finding container a8c585206c85c520a8c37a04845be0790ce5e61bbc3430a7f14f1943077cb96b: Status 404 returned error can't find the container with id a8c585206c85c520a8c37a04845be0790ce5e61bbc3430a7f14f1943077cb96b Jul 24 17:25:25.229591 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:25.229488 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" event={"ID":"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b","Type":"ContainerStarted","Data":"a8c585206c85c520a8c37a04845be0790ce5e61bbc3430a7f14f1943077cb96b"} Jul 24 17:25:28.243945 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:28.243908 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" event={"ID":"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b","Type":"ContainerStarted","Data":"accfa35e03cad82a53b1e9bf20b12a4268758cf6048838b2035fd8dddfc05b72"} Jul 24 17:25:29.249733 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:29.249699 2559 generic.go:356] "Generic (PLEG): container finished" podID="4b19c3e1-fd2f-4538-acf1-8013eec8ed9b" containerID="accfa35e03cad82a53b1e9bf20b12a4268758cf6048838b2035fd8dddfc05b72" exitCode=0 Jul 24 17:25:29.250097 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:29.249771 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" event={"ID":"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b","Type":"ContainerDied","Data":"accfa35e03cad82a53b1e9bf20b12a4268758cf6048838b2035fd8dddfc05b72"} Jul 24 17:25:30.256179 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:30.256143 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" event={"ID":"4b19c3e1-fd2f-4538-acf1-8013eec8ed9b","Type":"ContainerStarted","Data":"f14c9497cb8dabee377b6844411cfba76cc4100a713564c542d51ca888e6807b"} Jul 24 17:25:30.256582 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:30.256275 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:30.283163 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:30.283019 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" podStartSLOduration=1.801324951 podStartE2EDuration="6.283000053s" totalImagesPullingTime="3.44126551s" totalInitContainerRuntime="1.040409592s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:25:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:25:30.280784461 +0000 UTC m=+327.932675933" watchObservedRunningTime="2026-07-24 17:25:30.283000053 +0000 UTC m=+327.934891528" Jul 24 17:25:35.232550 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:35.232504 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-bdcf88f84-zx57k" Jul 24 17:25:41.261829 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:41.261791 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb" Jul 24 17:25:44.364591 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.364549 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6"] Jul 24 17:25:44.376279 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.376251 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6" Jul 24 17:25:44.379883 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.379846 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 17:25:44.380674 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.380653 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 17:25:44.380791 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.380714 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-65jr8\"" Jul 24 17:25:44.380791 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.380741 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 17:25:44.380985 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.380970 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 17:25:44.393727 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.393701 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6"] Jul 24 17:25:44.397398 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.397371 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba67174-6d4c-4e6d-91a7-2d64fffb91cf-tls-certs\") pod \"kube-auth-proxy-6bd989cf94-wrrd6\" (UID: \"1ba67174-6d4c-4e6d-91a7-2d64fffb91cf\") " pod="openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6" Jul 24 17:25:44.397551 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.397449 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8dff\" (UniqueName: \"kubernetes.io/projected/1ba67174-6d4c-4e6d-91a7-2d64fffb91cf-kube-api-access-f8dff\") pod \"kube-auth-proxy-6bd989cf94-wrrd6\" (UID: \"1ba67174-6d4c-4e6d-91a7-2d64fffb91cf\") " pod="openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6" Jul 24 17:25:44.397551 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.397498 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ba67174-6d4c-4e6d-91a7-2d64fffb91cf-tmp\") pod \"kube-auth-proxy-6bd989cf94-wrrd6\" (UID: \"1ba67174-6d4c-4e6d-91a7-2d64fffb91cf\") " pod="openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6" Jul 24 17:25:44.498126 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.498093 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8dff\" (UniqueName: \"kubernetes.io/projected/1ba67174-6d4c-4e6d-91a7-2d64fffb91cf-kube-api-access-f8dff\") pod \"kube-auth-proxy-6bd989cf94-wrrd6\" (UID: \"1ba67174-6d4c-4e6d-91a7-2d64fffb91cf\") " pod="openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6" Jul 24 17:25:44.498327 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.498150 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ba67174-6d4c-4e6d-91a7-2d64fffb91cf-tmp\") pod \"kube-auth-proxy-6bd989cf94-wrrd6\" (UID: \"1ba67174-6d4c-4e6d-91a7-2d64fffb91cf\") " pod="openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6" Jul 24 17:25:44.498327 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.498272 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba67174-6d4c-4e6d-91a7-2d64fffb91cf-tls-certs\") pod \"kube-auth-proxy-6bd989cf94-wrrd6\" (UID: \"1ba67174-6d4c-4e6d-91a7-2d64fffb91cf\") " pod="openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6" Jul 24 17:25:44.500470 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.500450 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ba67174-6d4c-4e6d-91a7-2d64fffb91cf-tmp\") pod \"kube-auth-proxy-6bd989cf94-wrrd6\" (UID: \"1ba67174-6d4c-4e6d-91a7-2d64fffb91cf\") " pod="openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6" Jul 24 17:25:44.500690 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.500673 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba67174-6d4c-4e6d-91a7-2d64fffb91cf-tls-certs\") pod \"kube-auth-proxy-6bd989cf94-wrrd6\" (UID: \"1ba67174-6d4c-4e6d-91a7-2d64fffb91cf\") " pod="openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6" Jul 24 17:25:44.509508 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.509473 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8dff\" (UniqueName: \"kubernetes.io/projected/1ba67174-6d4c-4e6d-91a7-2d64fffb91cf-kube-api-access-f8dff\") pod \"kube-auth-proxy-6bd989cf94-wrrd6\" (UID: \"1ba67174-6d4c-4e6d-91a7-2d64fffb91cf\") " pod="openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6" Jul 24 17:25:44.686452 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.686376 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6" Jul 24 17:25:44.821316 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:44.821282 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6"] Jul 24 17:25:44.824803 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:25:44.824761 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba67174_6d4c_4e6d_91a7_2d64fffb91cf.slice/crio-fe44a7987759a50d7835e9c9e25389d4846c9a24703378e7b2496e290f41a7a5 WatchSource:0}: Error finding container fe44a7987759a50d7835e9c9e25389d4846c9a24703378e7b2496e290f41a7a5: Status 404 returned error can't find the container with id fe44a7987759a50d7835e9c9e25389d4846c9a24703378e7b2496e290f41a7a5 Jul 24 17:25:45.313562 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:45.313497 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6" event={"ID":"1ba67174-6d4c-4e6d-91a7-2d64fffb91cf","Type":"ContainerStarted","Data":"fe44a7987759a50d7835e9c9e25389d4846c9a24703378e7b2496e290f41a7a5"} Jul 24 17:25:48.327427 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:48.327389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6" event={"ID":"1ba67174-6d4c-4e6d-91a7-2d64fffb91cf","Type":"ContainerStarted","Data":"763aaeb30f5d9b289fc991294d3b5d05edd1d4be24616f042940abee5b58cf96"} Jul 24 17:25:48.353032 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:25:48.352970 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6" podStartSLOduration=1.788453439 podStartE2EDuration="4.352952152s" totalImagesPullingTime="2.564498713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:25:48.351612047 +0000 UTC m=+346.003503520" watchObservedRunningTime="2026-07-24 17:25:48.352952152 +0000 UTC m=+346.004843655" Jul 24 17:28:27.428694 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.428654 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x79h6"] Jul 24 17:28:27.431987 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.431966 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" Jul 24 17:28:27.434705 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.434685 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 17:28:27.435597 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.435573 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-b2jbn\"" Jul 24 17:28:27.435597 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.435585 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 17:28:27.435823 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.435579 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 17:28:27.440425 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.440405 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x79h6"] Jul 24 17:28:27.498115 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.498083 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w85l\" (UniqueName: \"kubernetes.io/projected/ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc-kube-api-access-9w85l\") pod \"limitador-limitador-64c94fd64f-x79h6\" (UID: \"ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" Jul 24 17:28:27.498300 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.498155 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc-config-file\") pod \"limitador-limitador-64c94fd64f-x79h6\" (UID: \"ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" Jul 24 17:28:27.519451 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.519422 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x79h6"] Jul 24 17:28:27.599313 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.599282 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc-config-file\") pod \"limitador-limitador-64c94fd64f-x79h6\" (UID: \"ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" Jul 24 17:28:27.599490 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.599342 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w85l\" (UniqueName: \"kubernetes.io/projected/ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc-kube-api-access-9w85l\") pod \"limitador-limitador-64c94fd64f-x79h6\" (UID: \"ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" Jul 24 17:28:27.600001 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.599982 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc-config-file\") pod \"limitador-limitador-64c94fd64f-x79h6\" (UID: \"ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" Jul 24 17:28:27.608171 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.608140 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w85l\" (UniqueName: \"kubernetes.io/projected/ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc-kube-api-access-9w85l\") pod \"limitador-limitador-64c94fd64f-x79h6\" (UID: \"ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" Jul 24 17:28:27.743548 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.743433 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" Jul 24 17:28:27.872698 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.872675 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x79h6"] Jul 24 17:28:27.875361 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:28:27.875334 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5451f2_3f69_4ec4_9de1_9ca9c3d06ecc.slice/crio-2b6dd5e51c37784cddc95c227e9d7a06acca3d81f29c183b21f216db308e27bd WatchSource:0}: Error finding container 2b6dd5e51c37784cddc95c227e9d7a06acca3d81f29c183b21f216db308e27bd: Status 404 returned error can't find the container with id 2b6dd5e51c37784cddc95c227e9d7a06acca3d81f29c183b21f216db308e27bd Jul 24 17:28:27.884795 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:27.884768 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" event={"ID":"ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc","Type":"ContainerStarted","Data":"2b6dd5e51c37784cddc95c227e9d7a06acca3d81f29c183b21f216db308e27bd"} Jul 24 17:28:28.236156 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:28.236120 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-6bjsj"] Jul 24 17:28:28.240363 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:28.240340 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6bjsj" Jul 24 17:28:28.244103 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:28.244086 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zzl29\"" Jul 24 17:28:28.251789 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:28.251767 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-6bjsj"] Jul 24 17:28:28.406158 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:28.406127 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shgxf\" (UniqueName: \"kubernetes.io/projected/bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd-kube-api-access-shgxf\") pod \"authorino-58cd955675-6bjsj\" (UID: \"bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd\") " pod="kuadrant-system/authorino-58cd955675-6bjsj" Jul 24 17:28:28.507372 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:28.507290 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shgxf\" (UniqueName: \"kubernetes.io/projected/bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd-kube-api-access-shgxf\") pod \"authorino-58cd955675-6bjsj\" (UID: \"bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd\") " pod="kuadrant-system/authorino-58cd955675-6bjsj" Jul 24 17:28:28.527379 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:28.527323 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shgxf\" (UniqueName: \"kubernetes.io/projected/bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd-kube-api-access-shgxf\") pod \"authorino-58cd955675-6bjsj\" (UID: \"bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd\") " pod="kuadrant-system/authorino-58cd955675-6bjsj" Jul 24 17:28:28.550275 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:28.549983 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6bjsj" Jul 24 17:28:28.736104 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:28.736030 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-6bjsj"] Jul 24 17:28:28.743692 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:28:28.743658 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf4e4e19_60f3_4865_a0a3_2e8f4a7cffdd.slice/crio-735b8d39ba8ca36676b482878cb8ee8d2392cac78edf940a07f6b03de5ad3696 WatchSource:0}: Error finding container 735b8d39ba8ca36676b482878cb8ee8d2392cac78edf940a07f6b03de5ad3696: Status 404 returned error can't find the container with id 735b8d39ba8ca36676b482878cb8ee8d2392cac78edf940a07f6b03de5ad3696 Jul 24 17:28:28.890842 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:28.890806 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6bjsj" event={"ID":"bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd","Type":"ContainerStarted","Data":"735b8d39ba8ca36676b482878cb8ee8d2392cac78edf940a07f6b03de5ad3696"} Jul 24 17:28:30.901733 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:30.901694 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" event={"ID":"ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc","Type":"ContainerStarted","Data":"df541ace27ddc5619c5457b0f74d057ecb8804eb95b619ceb112e4208905012a"} Jul 24 17:28:30.902159 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:30.901818 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" Jul 24 17:28:30.920003 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:30.919650 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" podStartSLOduration=1.040761262 podStartE2EDuration="3.919624223s" totalImagesPullingTime="2.878862961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:28:30.918997883 +0000 UTC m=+508.570889356" watchObservedRunningTime="2026-07-24 17:28:30.919624223 +0000 UTC m=+508.571515698" Jul 24 17:28:32.479841 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:32.479768 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-6bjsj"] Jul 24 17:28:32.912030 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:32.911999 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6bjsj" event={"ID":"bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd","Type":"ContainerStarted","Data":"6dd88a3ec8b265dd9e8b1240f871da2d667789b3037bdac482f1ec4156288ee0"} Jul 24 17:28:32.912192 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:32.912055 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-6bjsj" podUID="bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd" containerName="authorino" containerID="cri-o://6dd88a3ec8b265dd9e8b1240f871da2d667789b3037bdac482f1ec4156288ee0" gracePeriod=30 Jul 24 17:28:32.932373 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:32.932167 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-6bjsj" podStartSLOduration=1.122662177 podStartE2EDuration="4.932151622s" totalImagesPullingTime="3.809489445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:28:32.930079962 +0000 UTC m=+510.581971449" watchObservedRunningTime="2026-07-24 17:28:32.932151622 +0000 UTC m=+510.584043095" Jul 24 17:28:33.152832 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:33.152802 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6bjsj" Jul 24 17:28:33.249826 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:33.249734 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shgxf\" (UniqueName: \"kubernetes.io/projected/bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd-kube-api-access-shgxf\") pod \"bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd\" (UID: \"bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd\") " Jul 24 17:28:33.252043 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:33.252016 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd-kube-api-access-shgxf" (OuterVolumeSpecName: "kube-api-access-shgxf") pod "bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd" (UID: "bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd"). InnerVolumeSpecName "kube-api-access-shgxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:28:33.351080 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:33.351043 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shgxf\" (UniqueName: \"kubernetes.io/projected/bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd-kube-api-access-shgxf\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:28:33.916504 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:33.916469 2559 generic.go:356] "Generic (PLEG): container finished" podID="bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd" containerID="6dd88a3ec8b265dd9e8b1240f871da2d667789b3037bdac482f1ec4156288ee0" exitCode=0 Jul 24 17:28:33.916960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:33.916533 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6bjsj" Jul 24 17:28:33.916960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:33.916560 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6bjsj" event={"ID":"bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd","Type":"ContainerDied","Data":"6dd88a3ec8b265dd9e8b1240f871da2d667789b3037bdac482f1ec4156288ee0"} Jul 24 17:28:33.916960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:33.916606 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6bjsj" event={"ID":"bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd","Type":"ContainerDied","Data":"735b8d39ba8ca36676b482878cb8ee8d2392cac78edf940a07f6b03de5ad3696"} Jul 24 17:28:33.916960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:33.916624 2559 scope.go:117] "RemoveContainer" containerID="6dd88a3ec8b265dd9e8b1240f871da2d667789b3037bdac482f1ec4156288ee0" Jul 24 17:28:33.924880 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:33.924865 2559 scope.go:117] "RemoveContainer" containerID="6dd88a3ec8b265dd9e8b1240f871da2d667789b3037bdac482f1ec4156288ee0" Jul 24 17:28:33.925122 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:28:33.925105 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd88a3ec8b265dd9e8b1240f871da2d667789b3037bdac482f1ec4156288ee0\": container with ID starting with 6dd88a3ec8b265dd9e8b1240f871da2d667789b3037bdac482f1ec4156288ee0 not found: ID does not exist" containerID="6dd88a3ec8b265dd9e8b1240f871da2d667789b3037bdac482f1ec4156288ee0" Jul 24 17:28:33.925176 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:33.925131 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd88a3ec8b265dd9e8b1240f871da2d667789b3037bdac482f1ec4156288ee0"} err="failed to get container status \"6dd88a3ec8b265dd9e8b1240f871da2d667789b3037bdac482f1ec4156288ee0\": rpc error: code = NotFound desc = could not find container \"6dd88a3ec8b265dd9e8b1240f871da2d667789b3037bdac482f1ec4156288ee0\": container with ID starting with 6dd88a3ec8b265dd9e8b1240f871da2d667789b3037bdac482f1ec4156288ee0 not found: ID does not exist" Jul 24 17:28:33.938377 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:33.938338 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-6bjsj"] Jul 24 17:28:33.939824 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:33.939802 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-6bjsj"] Jul 24 17:28:34.941283 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:34.941248 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd" path="/var/lib/kubelet/pods/bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd/volumes" Jul 24 17:28:41.907481 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:41.907451 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" Jul 24 17:28:42.864427 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:42.864391 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x79h6"] Jul 24 17:28:42.864622 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:42.864599 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" podUID="ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc" containerName="limitador" containerID="cri-o://df541ace27ddc5619c5457b0f74d057ecb8804eb95b619ceb112e4208905012a" gracePeriod=30 Jul 24 17:28:43.411859 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.411835 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" Jul 24 17:28:43.538279 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.538245 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9w85l\" (UniqueName: \"kubernetes.io/projected/ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc-kube-api-access-9w85l\") pod \"ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc\" (UID: \"ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc\") " Jul 24 17:28:43.538448 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.538314 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc-config-file\") pod \"ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc\" (UID: \"ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc\") " Jul 24 17:28:43.538733 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.538707 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc-config-file" (OuterVolumeSpecName: "config-file") pod "ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc" (UID: "ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:28:43.540600 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.540579 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc-kube-api-access-9w85l" (OuterVolumeSpecName: "kube-api-access-9w85l") pod "ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc" (UID: "ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc"). InnerVolumeSpecName "kube-api-access-9w85l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:28:43.639146 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.639108 2559 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc-config-file\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:28:43.639146 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.639138 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w85l\" (UniqueName: \"kubernetes.io/projected/ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc-kube-api-access-9w85l\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:28:43.953087 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.953005 2559 generic.go:356] "Generic (PLEG): container finished" podID="ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc" containerID="df541ace27ddc5619c5457b0f74d057ecb8804eb95b619ceb112e4208905012a" exitCode=0 Jul 24 17:28:43.953087 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.953053 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" Jul 24 17:28:43.953256 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.953088 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" event={"ID":"ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc","Type":"ContainerDied","Data":"df541ace27ddc5619c5457b0f74d057ecb8804eb95b619ceb112e4208905012a"} Jul 24 17:28:43.953256 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.953139 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x79h6" event={"ID":"ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc","Type":"ContainerDied","Data":"2b6dd5e51c37784cddc95c227e9d7a06acca3d81f29c183b21f216db308e27bd"} Jul 24 17:28:43.953256 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.953163 2559 scope.go:117] "RemoveContainer" containerID="df541ace27ddc5619c5457b0f74d057ecb8804eb95b619ceb112e4208905012a" Jul 24 17:28:43.961695 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.961676 2559 scope.go:117] "RemoveContainer" containerID="df541ace27ddc5619c5457b0f74d057ecb8804eb95b619ceb112e4208905012a" Jul 24 17:28:43.961946 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:28:43.961926 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df541ace27ddc5619c5457b0f74d057ecb8804eb95b619ceb112e4208905012a\": container with ID starting with df541ace27ddc5619c5457b0f74d057ecb8804eb95b619ceb112e4208905012a not found: ID does not exist" containerID="df541ace27ddc5619c5457b0f74d057ecb8804eb95b619ceb112e4208905012a" Jul 24 17:28:43.961997 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.961955 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df541ace27ddc5619c5457b0f74d057ecb8804eb95b619ceb112e4208905012a"} err="failed to get container status \"df541ace27ddc5619c5457b0f74d057ecb8804eb95b619ceb112e4208905012a\": rpc error: code = NotFound desc = could not find container \"df541ace27ddc5619c5457b0f74d057ecb8804eb95b619ceb112e4208905012a\": container with ID starting with df541ace27ddc5619c5457b0f74d057ecb8804eb95b619ceb112e4208905012a not found: ID does not exist" Jul 24 17:28:43.973490 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.973467 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x79h6"] Jul 24 17:28:43.979121 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:43.979100 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x79h6"] Jul 24 17:28:44.942087 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:44.942056 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc" path="/var/lib/kubelet/pods/ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc/volumes" Jul 24 17:28:49.316210 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.316172 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk"] Jul 24 17:28:49.316615 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.316599 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc" containerName="limitador" Jul 24 17:28:49.316669 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.316620 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc" containerName="limitador" Jul 24 17:28:49.316669 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.316632 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd" containerName="authorino" Jul 24 17:28:49.316669 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.316641 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd" containerName="authorino" Jul 24 17:28:49.316760 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.316737 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf4e4e19-60f3-4865-a0a3-2e8f4a7cffdd" containerName="authorino" Jul 24 17:28:49.316760 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.316751 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae5451f2-3f69-4ec4-9de1-9ca9c3d06ecc" containerName="limitador" Jul 24 17:28:49.322275 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.322256 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk" Jul 24 17:28:49.326226 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.326204 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 17:28:49.326341 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.326204 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 17:28:49.328189 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.328168 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 17:28:49.328942 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.328786 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-m9hh4\"" Jul 24 17:28:49.330107 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.330086 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk"] Jul 24 17:28:49.380785 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.380754 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml7bg\" (UniqueName: \"kubernetes.io/projected/2943ea2f-1547-4a19-8a27-285093d45c82-kube-api-access-ml7bg\") pod \"postgres-69fb4594cf-xvxlk\" (UID: \"2943ea2f-1547-4a19-8a27-285093d45c82\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk" Jul 24 17:28:49.380952 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.380811 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2943ea2f-1547-4a19-8a27-285093d45c82-data\") pod \"postgres-69fb4594cf-xvxlk\" (UID: \"2943ea2f-1547-4a19-8a27-285093d45c82\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk" Jul 24 17:28:49.481603 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.481573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml7bg\" (UniqueName: \"kubernetes.io/projected/2943ea2f-1547-4a19-8a27-285093d45c82-kube-api-access-ml7bg\") pod \"postgres-69fb4594cf-xvxlk\" (UID: \"2943ea2f-1547-4a19-8a27-285093d45c82\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk" Jul 24 17:28:49.481747 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.481643 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2943ea2f-1547-4a19-8a27-285093d45c82-data\") pod \"postgres-69fb4594cf-xvxlk\" (UID: \"2943ea2f-1547-4a19-8a27-285093d45c82\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk" Jul 24 17:28:49.482008 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.481993 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2943ea2f-1547-4a19-8a27-285093d45c82-data\") pod \"postgres-69fb4594cf-xvxlk\" (UID: \"2943ea2f-1547-4a19-8a27-285093d45c82\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk" Jul 24 17:28:49.491936 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.491913 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml7bg\" (UniqueName: \"kubernetes.io/projected/2943ea2f-1547-4a19-8a27-285093d45c82-kube-api-access-ml7bg\") pod \"postgres-69fb4594cf-xvxlk\" (UID: \"2943ea2f-1547-4a19-8a27-285093d45c82\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk" Jul 24 17:28:49.632360 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.632272 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk" Jul 24 17:28:49.759647 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.759617 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk"] Jul 24 17:28:49.762818 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:28:49.762786 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2943ea2f_1547_4a19_8a27_285093d45c82.slice/crio-a5746385c9a9123e80274da8fdeb5c8426409332e1a6f2236db71866e790b4c4 WatchSource:0}: Error finding container a5746385c9a9123e80274da8fdeb5c8426409332e1a6f2236db71866e790b4c4: Status 404 returned error can't find the container with id a5746385c9a9123e80274da8fdeb5c8426409332e1a6f2236db71866e790b4c4 Jul 24 17:28:49.982032 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:49.981942 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk" event={"ID":"2943ea2f-1547-4a19-8a27-285093d45c82","Type":"ContainerStarted","Data":"a5746385c9a9123e80274da8fdeb5c8426409332e1a6f2236db71866e790b4c4"} Jul 24 17:28:56.009862 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:56.009832 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk" event={"ID":"2943ea2f-1547-4a19-8a27-285093d45c82","Type":"ContainerStarted","Data":"cccbaaf6ed86e0cc33fb26cbc24982d92c210080e9160cc542007195284a322f"} Jul 24 17:28:56.010256 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:56.009968 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk" Jul 24 17:28:56.026900 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:28:56.026841 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk" podStartSLOduration=1.694866389 podStartE2EDuration="7.026823134s" totalImagesPullingTime="5.331956745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:28:56.026344254 +0000 UTC m=+533.678235728" watchObservedRunningTime="2026-07-24 17:28:56.026823134 +0000 UTC m=+533.678714607" Jul 24 17:29:02.042365 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.042336 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xvxlk" Jul 24 17:29:02.594925 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.594890 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hf2hr"] Jul 24 17:29:02.598233 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.598218 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hf2hr" Jul 24 17:29:02.601149 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.601130 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 17:29:02.601239 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.601130 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 17:29:02.602209 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.602194 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zzl29\"" Jul 24 17:29:02.606766 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.606746 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hf2hr"] Jul 24 17:29:02.686354 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.686306 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkm9d\" (UniqueName: \"kubernetes.io/projected/3a2c719f-e1d3-41e5-a5ab-5727f2404a60-kube-api-access-vkm9d\") pod \"authorino-5d4b67c446-hf2hr\" (UID: \"3a2c719f-e1d3-41e5-a5ab-5727f2404a60\") " pod="kuadrant-system/authorino-5d4b67c446-hf2hr" Jul 24 17:29:02.787351 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.787321 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkm9d\" (UniqueName: \"kubernetes.io/projected/3a2c719f-e1d3-41e5-a5ab-5727f2404a60-kube-api-access-vkm9d\") pod \"authorino-5d4b67c446-hf2hr\" (UID: \"3a2c719f-e1d3-41e5-a5ab-5727f2404a60\") " pod="kuadrant-system/authorino-5d4b67c446-hf2hr" Jul 24 17:29:02.799012 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.798986 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 17:29:02.810247 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.810228 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 17:29:02.819086 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.819063 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkm9d\" (UniqueName: \"kubernetes.io/projected/3a2c719f-e1d3-41e5-a5ab-5727f2404a60-kube-api-access-vkm9d\") pod \"authorino-5d4b67c446-hf2hr\" (UID: \"3a2c719f-e1d3-41e5-a5ab-5727f2404a60\") " pod="kuadrant-system/authorino-5d4b67c446-hf2hr" Jul 24 17:29:02.873022 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.872943 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hf2hr"] Jul 24 17:29:02.875633 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.875615 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zzl29\"" Jul 24 17:29:02.883941 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.883925 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hf2hr" Jul 24 17:29:02.907216 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.907193 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5bdfb766f5-dq7rj"] Jul 24 17:29:02.910653 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.910634 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bdfb766f5-dq7rj" Jul 24 17:29:02.922250 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.922225 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bdfb766f5-dq7rj"] Jul 24 17:29:02.989131 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:02.989097 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f85wl\" (UniqueName: \"kubernetes.io/projected/3d7cd98e-8b0b-4670-9fd0-6870052a5e6e-kube-api-access-f85wl\") pod \"authorino-5bdfb766f5-dq7rj\" (UID: \"3d7cd98e-8b0b-4670-9fd0-6870052a5e6e\") " pod="kuadrant-system/authorino-5bdfb766f5-dq7rj" Jul 24 17:29:03.022596 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:03.022562 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hf2hr"] Jul 24 17:29:03.026069 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:29:03.026037 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a2c719f_e1d3_41e5_a5ab_5727f2404a60.slice/crio-4a24ce368c10ab9a9393814c53a64ce62abbf7d3a4e96124d8fd1d3a5d9dc10f WatchSource:0}: Error finding container 4a24ce368c10ab9a9393814c53a64ce62abbf7d3a4e96124d8fd1d3a5d9dc10f: Status 404 returned error can't find the container with id 4a24ce368c10ab9a9393814c53a64ce62abbf7d3a4e96124d8fd1d3a5d9dc10f Jul 24 17:29:03.037654 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:03.037628 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-hf2hr" event={"ID":"3a2c719f-e1d3-41e5-a5ab-5727f2404a60","Type":"ContainerStarted","Data":"4a24ce368c10ab9a9393814c53a64ce62abbf7d3a4e96124d8fd1d3a5d9dc10f"} Jul 24 17:29:03.090268 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:03.090229 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f85wl\" (UniqueName: \"kubernetes.io/projected/3d7cd98e-8b0b-4670-9fd0-6870052a5e6e-kube-api-access-f85wl\") pod \"authorino-5bdfb766f5-dq7rj\" (UID: \"3d7cd98e-8b0b-4670-9fd0-6870052a5e6e\") " pod="kuadrant-system/authorino-5bdfb766f5-dq7rj" Jul 24 17:29:03.100799 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:03.100771 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f85wl\" (UniqueName: \"kubernetes.io/projected/3d7cd98e-8b0b-4670-9fd0-6870052a5e6e-kube-api-access-f85wl\") pod \"authorino-5bdfb766f5-dq7rj\" (UID: \"3d7cd98e-8b0b-4670-9fd0-6870052a5e6e\") " pod="kuadrant-system/authorino-5bdfb766f5-dq7rj" Jul 24 17:29:03.150589 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:03.150501 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5bdfb766f5-dq7rj"] Jul 24 17:29:03.150766 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:03.150754 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bdfb766f5-dq7rj" Jul 24 17:29:03.278578 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:03.278546 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5bdfb766f5-dq7rj"] Jul 24 17:29:03.281932 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:29:03.281908 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d7cd98e_8b0b_4670_9fd0_6870052a5e6e.slice/crio-60ab158bff1c3bdc9071af40a9aa5a3c510fb2d9473258ee64b7459ea5aa9bd8 WatchSource:0}: Error finding container 60ab158bff1c3bdc9071af40a9aa5a3c510fb2d9473258ee64b7459ea5aa9bd8: Status 404 returned error can't find the container with id 60ab158bff1c3bdc9071af40a9aa5a3c510fb2d9473258ee64b7459ea5aa9bd8 Jul 24 17:29:04.042509 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.042474 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-hf2hr" event={"ID":"3a2c719f-e1d3-41e5-a5ab-5727f2404a60","Type":"ContainerStarted","Data":"13b32dfb265d3c25349e637b23aa22084c10abd7dd3de445525e1b9969f9c089"} Jul 24 17:29:04.042725 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.042566 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-hf2hr" podUID="3a2c719f-e1d3-41e5-a5ab-5727f2404a60" containerName="authorino" containerID="cri-o://13b32dfb265d3c25349e637b23aa22084c10abd7dd3de445525e1b9969f9c089" gracePeriod=30 Jul 24 17:29:04.044180 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.044157 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bdfb766f5-dq7rj" event={"ID":"3d7cd98e-8b0b-4670-9fd0-6870052a5e6e","Type":"ContainerStarted","Data":"83afe13846874aaa9adaa60535311d92780789cab7005dfb4e98922e6d9de983"} Jul 24 17:29:04.044288 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.044187 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bdfb766f5-dq7rj" event={"ID":"3d7cd98e-8b0b-4670-9fd0-6870052a5e6e","Type":"ContainerStarted","Data":"60ab158bff1c3bdc9071af40a9aa5a3c510fb2d9473258ee64b7459ea5aa9bd8"} Jul 24 17:29:04.044288 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.044211 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5bdfb766f5-dq7rj" podUID="3d7cd98e-8b0b-4670-9fd0-6870052a5e6e" containerName="authorino" containerID="cri-o://83afe13846874aaa9adaa60535311d92780789cab7005dfb4e98922e6d9de983" gracePeriod=30 Jul 24 17:29:04.061060 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.061008 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-hf2hr" podStartSLOduration=1.704476055 podStartE2EDuration="2.060987328s" totalImagesPullingTime="356.511273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:29:04.058809573 +0000 UTC m=+541.710701048" watchObservedRunningTime="2026-07-24 17:29:04.060987328 +0000 UTC m=+541.712878802" Jul 24 17:29:04.077817 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.077774 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5bdfb766f5-dq7rj" podStartSLOduration=1.658823086 podStartE2EDuration="2.077759061s" totalImagesPullingTime="418.935975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:29:04.076588418 +0000 UTC m=+541.728479903" watchObservedRunningTime="2026-07-24 17:29:04.077759061 +0000 UTC m=+541.729650546" Jul 24 17:29:04.085710 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:29:04.085684 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d7cd98e_8b0b_4670_9fd0_6870052a5e6e.slice/crio-conmon-83afe13846874aaa9adaa60535311d92780789cab7005dfb4e98922e6d9de983.scope\": RecentStats: unable to find data in memory cache]" Jul 24 17:29:04.300145 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.300124 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bdfb766f5-dq7rj" Jul 24 17:29:04.302668 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.302654 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hf2hr" Jul 24 17:29:04.400484 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.400449 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f85wl\" (UniqueName: \"kubernetes.io/projected/3d7cd98e-8b0b-4670-9fd0-6870052a5e6e-kube-api-access-f85wl\") pod \"3d7cd98e-8b0b-4670-9fd0-6870052a5e6e\" (UID: \"3d7cd98e-8b0b-4670-9fd0-6870052a5e6e\") " Jul 24 17:29:04.400484 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.400493 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkm9d\" (UniqueName: \"kubernetes.io/projected/3a2c719f-e1d3-41e5-a5ab-5727f2404a60-kube-api-access-vkm9d\") pod \"3a2c719f-e1d3-41e5-a5ab-5727f2404a60\" (UID: \"3a2c719f-e1d3-41e5-a5ab-5727f2404a60\") " Jul 24 17:29:04.402643 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.402615 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2c719f-e1d3-41e5-a5ab-5727f2404a60-kube-api-access-vkm9d" (OuterVolumeSpecName: "kube-api-access-vkm9d") pod "3a2c719f-e1d3-41e5-a5ab-5727f2404a60" (UID: "3a2c719f-e1d3-41e5-a5ab-5727f2404a60"). InnerVolumeSpecName "kube-api-access-vkm9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:29:04.402643 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.402629 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7cd98e-8b0b-4670-9fd0-6870052a5e6e-kube-api-access-f85wl" (OuterVolumeSpecName: "kube-api-access-f85wl") pod "3d7cd98e-8b0b-4670-9fd0-6870052a5e6e" (UID: "3d7cd98e-8b0b-4670-9fd0-6870052a5e6e"). InnerVolumeSpecName "kube-api-access-f85wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:29:04.501840 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.501803 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f85wl\" (UniqueName: \"kubernetes.io/projected/3d7cd98e-8b0b-4670-9fd0-6870052a5e6e-kube-api-access-f85wl\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:29:04.501840 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:04.501832 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkm9d\" (UniqueName: \"kubernetes.io/projected/3a2c719f-e1d3-41e5-a5ab-5727f2404a60-kube-api-access-vkm9d\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:29:05.049210 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.049174 2559 generic.go:356] "Generic (PLEG): container finished" podID="3a2c719f-e1d3-41e5-a5ab-5727f2404a60" containerID="13b32dfb265d3c25349e637b23aa22084c10abd7dd3de445525e1b9969f9c089" exitCode=0 Jul 24 17:29:05.049360 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.049218 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hf2hr" Jul 24 17:29:05.049360 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.049245 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-hf2hr" event={"ID":"3a2c719f-e1d3-41e5-a5ab-5727f2404a60","Type":"ContainerDied","Data":"13b32dfb265d3c25349e637b23aa22084c10abd7dd3de445525e1b9969f9c089"} Jul 24 17:29:05.049360 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.049277 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-hf2hr" event={"ID":"3a2c719f-e1d3-41e5-a5ab-5727f2404a60","Type":"ContainerDied","Data":"4a24ce368c10ab9a9393814c53a64ce62abbf7d3a4e96124d8fd1d3a5d9dc10f"} Jul 24 17:29:05.049360 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.049299 2559 scope.go:117] "RemoveContainer" containerID="13b32dfb265d3c25349e637b23aa22084c10abd7dd3de445525e1b9969f9c089" Jul 24 17:29:05.050648 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.050466 2559 generic.go:356] "Generic (PLEG): container finished" podID="3d7cd98e-8b0b-4670-9fd0-6870052a5e6e" containerID="83afe13846874aaa9adaa60535311d92780789cab7005dfb4e98922e6d9de983" exitCode=0 Jul 24 17:29:05.050648 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.050528 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bdfb766f5-dq7rj" Jul 24 17:29:05.050648 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.050543 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bdfb766f5-dq7rj" event={"ID":"3d7cd98e-8b0b-4670-9fd0-6870052a5e6e","Type":"ContainerDied","Data":"83afe13846874aaa9adaa60535311d92780789cab7005dfb4e98922e6d9de983"} Jul 24 17:29:05.050648 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.050573 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bdfb766f5-dq7rj" event={"ID":"3d7cd98e-8b0b-4670-9fd0-6870052a5e6e","Type":"ContainerDied","Data":"60ab158bff1c3bdc9071af40a9aa5a3c510fb2d9473258ee64b7459ea5aa9bd8"} Jul 24 17:29:05.058002 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.057984 2559 scope.go:117] "RemoveContainer" containerID="13b32dfb265d3c25349e637b23aa22084c10abd7dd3de445525e1b9969f9c089" Jul 24 17:29:05.058249 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:29:05.058234 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b32dfb265d3c25349e637b23aa22084c10abd7dd3de445525e1b9969f9c089\": container with ID starting with 13b32dfb265d3c25349e637b23aa22084c10abd7dd3de445525e1b9969f9c089 not found: ID does not exist" containerID="13b32dfb265d3c25349e637b23aa22084c10abd7dd3de445525e1b9969f9c089" Jul 24 17:29:05.058302 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.058259 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b32dfb265d3c25349e637b23aa22084c10abd7dd3de445525e1b9969f9c089"} err="failed to get container status \"13b32dfb265d3c25349e637b23aa22084c10abd7dd3de445525e1b9969f9c089\": rpc error: code = NotFound desc = could not find container \"13b32dfb265d3c25349e637b23aa22084c10abd7dd3de445525e1b9969f9c089\": container with ID starting with 13b32dfb265d3c25349e637b23aa22084c10abd7dd3de445525e1b9969f9c089 not found: ID does not exist" Jul 24 17:29:05.058302 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.058275 2559 scope.go:117] "RemoveContainer" containerID="83afe13846874aaa9adaa60535311d92780789cab7005dfb4e98922e6d9de983" Jul 24 17:29:05.065604 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.065585 2559 scope.go:117] "RemoveContainer" containerID="83afe13846874aaa9adaa60535311d92780789cab7005dfb4e98922e6d9de983" Jul 24 17:29:05.065879 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:29:05.065857 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83afe13846874aaa9adaa60535311d92780789cab7005dfb4e98922e6d9de983\": container with ID starting with 83afe13846874aaa9adaa60535311d92780789cab7005dfb4e98922e6d9de983 not found: ID does not exist" containerID="83afe13846874aaa9adaa60535311d92780789cab7005dfb4e98922e6d9de983" Jul 24 17:29:05.065943 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.065886 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83afe13846874aaa9adaa60535311d92780789cab7005dfb4e98922e6d9de983"} err="failed to get container status \"83afe13846874aaa9adaa60535311d92780789cab7005dfb4e98922e6d9de983\": rpc error: code = NotFound desc = could not find container \"83afe13846874aaa9adaa60535311d92780789cab7005dfb4e98922e6d9de983\": container with ID starting with 83afe13846874aaa9adaa60535311d92780789cab7005dfb4e98922e6d9de983 not found: ID does not exist" Jul 24 17:29:05.069849 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.069829 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hf2hr"] Jul 24 17:29:05.076558 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.076487 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hf2hr"] Jul 24 17:29:05.087486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.087464 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5bdfb766f5-dq7rj"] Jul 24 17:29:05.089748 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:05.089724 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5bdfb766f5-dq7rj"] Jul 24 17:29:06.942920 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:06.942889 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a2c719f-e1d3-41e5-a5ab-5727f2404a60" path="/var/lib/kubelet/pods/3a2c719f-e1d3-41e5-a5ab-5727f2404a60/volumes" Jul 24 17:29:06.943304 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:06.943191 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d7cd98e-8b0b-4670-9fd0-6870052a5e6e" path="/var/lib/kubelet/pods/3d7cd98e-8b0b-4670-9fd0-6870052a5e6e/volumes" Jul 24 17:29:09.363212 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.363170 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56dc868d8b-zw7k7"] Jul 24 17:29:09.363680 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.363601 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7cd98e-8b0b-4670-9fd0-6870052a5e6e" containerName="authorino" Jul 24 17:29:09.363680 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.363614 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7cd98e-8b0b-4670-9fd0-6870052a5e6e" containerName="authorino" Jul 24 17:29:09.363680 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.363622 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a2c719f-e1d3-41e5-a5ab-5727f2404a60" containerName="authorino" Jul 24 17:29:09.363680 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.363628 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a2c719f-e1d3-41e5-a5ab-5727f2404a60" containerName="authorino" Jul 24 17:29:09.363809 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.363691 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7cd98e-8b0b-4670-9fd0-6870052a5e6e" containerName="authorino" Jul 24 17:29:09.363809 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.363699 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a2c719f-e1d3-41e5-a5ab-5727f2404a60" containerName="authorino" Jul 24 17:29:09.368098 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.368080 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56dc868d8b-zw7k7" Jul 24 17:29:09.371927 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.371907 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 17:29:09.372210 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.372182 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 17:29:09.372291 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.372276 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9hhf2\"" Jul 24 17:29:09.384611 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.384591 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56dc868d8b-zw7k7"] Jul 24 17:29:09.442316 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.442283 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hprrs\" (UniqueName: \"kubernetes.io/projected/d5fc1805-d654-4e40-b1ea-276d830b09dc-kube-api-access-hprrs\") pod \"maas-controller-56dc868d8b-zw7k7\" (UID: \"d5fc1805-d654-4e40-b1ea-276d830b09dc\") " pod="opendatahub/maas-controller-56dc868d8b-zw7k7" Jul 24 17:29:09.442475 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.442337 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5fc1805-d654-4e40-b1ea-276d830b09dc-webhook-cert\") pod \"maas-controller-56dc868d8b-zw7k7\" (UID: \"d5fc1805-d654-4e40-b1ea-276d830b09dc\") " pod="opendatahub/maas-controller-56dc868d8b-zw7k7" Jul 24 17:29:09.532885 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.532854 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-2458k"] Jul 24 17:29:09.536123 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.536107 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bbcff6f4-2458k" Jul 24 17:29:09.542879 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.542857 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hprrs\" (UniqueName: \"kubernetes.io/projected/d5fc1805-d654-4e40-b1ea-276d830b09dc-kube-api-access-hprrs\") pod \"maas-controller-56dc868d8b-zw7k7\" (UID: \"d5fc1805-d654-4e40-b1ea-276d830b09dc\") " pod="opendatahub/maas-controller-56dc868d8b-zw7k7" Jul 24 17:29:09.542978 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.542902 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5fc1805-d654-4e40-b1ea-276d830b09dc-webhook-cert\") pod \"maas-controller-56dc868d8b-zw7k7\" (UID: \"d5fc1805-d654-4e40-b1ea-276d830b09dc\") " pod="opendatahub/maas-controller-56dc868d8b-zw7k7" Jul 24 17:29:09.543061 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:29:09.542990 2559 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 24 17:29:09.543061 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:29:09.543036 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5fc1805-d654-4e40-b1ea-276d830b09dc-webhook-cert podName:d5fc1805-d654-4e40-b1ea-276d830b09dc nodeName:}" failed. No retries permitted until 2026-07-24 17:29:10.043022746 +0000 UTC m=+547.694914196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/d5fc1805-d654-4e40-b1ea-276d830b09dc-webhook-cert") pod "maas-controller-56dc868d8b-zw7k7" (UID: "d5fc1805-d654-4e40-b1ea-276d830b09dc") : secret "maas-controller-webhook-cert" not found Jul 24 17:29:09.556122 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.556100 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-2458k"] Jul 24 17:29:09.561200 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.561179 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hprrs\" (UniqueName: \"kubernetes.io/projected/d5fc1805-d654-4e40-b1ea-276d830b09dc-kube-api-access-hprrs\") pod \"maas-controller-56dc868d8b-zw7k7\" (UID: \"d5fc1805-d654-4e40-b1ea-276d830b09dc\") " pod="opendatahub/maas-controller-56dc868d8b-zw7k7" Jul 24 17:29:09.644172 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.644087 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmjl\" (UniqueName: \"kubernetes.io/projected/89cc0d44-2d10-4f6a-8862-afd36d054376-kube-api-access-xkmjl\") pod \"maas-controller-56bbcff6f4-2458k\" (UID: \"89cc0d44-2d10-4f6a-8862-afd36d054376\") " pod="opendatahub/maas-controller-56bbcff6f4-2458k" Jul 24 17:29:09.644172 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.644129 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89cc0d44-2d10-4f6a-8862-afd36d054376-webhook-cert\") pod \"maas-controller-56bbcff6f4-2458k\" (UID: \"89cc0d44-2d10-4f6a-8862-afd36d054376\") " pod="opendatahub/maas-controller-56bbcff6f4-2458k" Jul 24 17:29:09.745044 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.745008 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmjl\" (UniqueName: \"kubernetes.io/projected/89cc0d44-2d10-4f6a-8862-afd36d054376-kube-api-access-xkmjl\") pod \"maas-controller-56bbcff6f4-2458k\" (UID: \"89cc0d44-2d10-4f6a-8862-afd36d054376\") " pod="opendatahub/maas-controller-56bbcff6f4-2458k" Jul 24 17:29:09.745178 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.745048 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89cc0d44-2d10-4f6a-8862-afd36d054376-webhook-cert\") pod \"maas-controller-56bbcff6f4-2458k\" (UID: \"89cc0d44-2d10-4f6a-8862-afd36d054376\") " pod="opendatahub/maas-controller-56bbcff6f4-2458k" Jul 24 17:29:09.747477 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.747453 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89cc0d44-2d10-4f6a-8862-afd36d054376-webhook-cert\") pod \"maas-controller-56bbcff6f4-2458k\" (UID: \"89cc0d44-2d10-4f6a-8862-afd36d054376\") " pod="opendatahub/maas-controller-56bbcff6f4-2458k" Jul 24 17:29:09.754718 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.754692 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmjl\" (UniqueName: \"kubernetes.io/projected/89cc0d44-2d10-4f6a-8862-afd36d054376-kube-api-access-xkmjl\") pod \"maas-controller-56bbcff6f4-2458k\" (UID: \"89cc0d44-2d10-4f6a-8862-afd36d054376\") " pod="opendatahub/maas-controller-56bbcff6f4-2458k" Jul 24 17:29:09.845581 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.845548 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bbcff6f4-2458k" Jul 24 17:29:09.981128 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:09.981094 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-2458k"] Jul 24 17:29:09.983876 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:29:09.983855 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cc0d44_2d10_4f6a_8862_afd36d054376.slice/crio-ce3939690cc01bb0d042b15ebed288f6df89ba0295edcb669390eb7790ae63cd WatchSource:0}: Error finding container ce3939690cc01bb0d042b15ebed288f6df89ba0295edcb669390eb7790ae63cd: Status 404 returned error can't find the container with id ce3939690cc01bb0d042b15ebed288f6df89ba0295edcb669390eb7790ae63cd Jul 24 17:29:10.047401 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:10.047363 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5fc1805-d654-4e40-b1ea-276d830b09dc-webhook-cert\") pod \"maas-controller-56dc868d8b-zw7k7\" (UID: \"d5fc1805-d654-4e40-b1ea-276d830b09dc\") " pod="opendatahub/maas-controller-56dc868d8b-zw7k7" Jul 24 17:29:10.049950 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:10.049931 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5fc1805-d654-4e40-b1ea-276d830b09dc-webhook-cert\") pod \"maas-controller-56dc868d8b-zw7k7\" (UID: \"d5fc1805-d654-4e40-b1ea-276d830b09dc\") " pod="opendatahub/maas-controller-56dc868d8b-zw7k7" Jul 24 17:29:10.071895 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:10.071859 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bbcff6f4-2458k" event={"ID":"89cc0d44-2d10-4f6a-8862-afd36d054376","Type":"ContainerStarted","Data":"ce3939690cc01bb0d042b15ebed288f6df89ba0295edcb669390eb7790ae63cd"} Jul 24 17:29:10.277874 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:10.277845 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56dc868d8b-zw7k7" Jul 24 17:29:10.429012 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:10.428975 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56dc868d8b-zw7k7"] Jul 24 17:29:10.432312 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:29:10.432286 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fc1805_d654_4e40_b1ea_276d830b09dc.slice/crio-15f8c588f393f443a0af07000abee4e8477b5c0b347a0c085e4a62f25e43b795 WatchSource:0}: Error finding container 15f8c588f393f443a0af07000abee4e8477b5c0b347a0c085e4a62f25e43b795: Status 404 returned error can't find the container with id 15f8c588f393f443a0af07000abee4e8477b5c0b347a0c085e4a62f25e43b795 Jul 24 17:29:11.080614 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:11.080575 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56dc868d8b-zw7k7" event={"ID":"d5fc1805-d654-4e40-b1ea-276d830b09dc","Type":"ContainerStarted","Data":"15f8c588f393f443a0af07000abee4e8477b5c0b347a0c085e4a62f25e43b795"} Jul 24 17:29:13.091216 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:13.091173 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56dc868d8b-zw7k7" event={"ID":"d5fc1805-d654-4e40-b1ea-276d830b09dc","Type":"ContainerStarted","Data":"b7aba81f45b2cf22270cda09bbf7369049fa5188cfb1da09707f83733caf137f"} Jul 24 17:29:13.091666 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:13.091247 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56dc868d8b-zw7k7" Jul 24 17:29:13.092594 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:13.092572 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bbcff6f4-2458k" event={"ID":"89cc0d44-2d10-4f6a-8862-afd36d054376","Type":"ContainerStarted","Data":"db959458a2c948b75f18cfa12a999f04a3fffc610fc2878951f0f631422c4573"} Jul 24 17:29:13.092720 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:13.092706 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56bbcff6f4-2458k" Jul 24 17:29:13.115730 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:13.115682 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56dc868d8b-zw7k7" podStartSLOduration=1.8174823359999999 podStartE2EDuration="4.115668352s" totalImagesPullingTime="2.298186016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:29:13.115203025 +0000 UTC m=+550.767094498" watchObservedRunningTime="2026-07-24 17:29:13.115668352 +0000 UTC m=+550.767559825" Jul 24 17:29:13.135824 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:13.135772 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56bbcff6f4-2458k" podStartSLOduration=1.395393021 podStartE2EDuration="4.135757543s" totalImagesPullingTime="2.740364522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:29:13.135027637 +0000 UTC m=+550.786919109" watchObservedRunningTime="2026-07-24 17:29:13.135757543 +0000 UTC m=+550.787649015" Jul 24 17:29:24.099266 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:24.099238 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56dc868d8b-zw7k7" Jul 24 17:29:24.099639 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:24.099303 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56bbcff6f4-2458k" Jul 24 17:29:24.168776 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:24.168740 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56dc868d8b-zw7k7"] Jul 24 17:29:24.168987 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:24.168958 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-56dc868d8b-zw7k7" podUID="d5fc1805-d654-4e40-b1ea-276d830b09dc" containerName="manager" containerID="cri-o://b7aba81f45b2cf22270cda09bbf7369049fa5188cfb1da09707f83733caf137f" gracePeriod=30 Jul 24 17:29:24.414094 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:24.414068 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56dc868d8b-zw7k7" Jul 24 17:29:24.584227 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:24.584193 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hprrs\" (UniqueName: \"kubernetes.io/projected/d5fc1805-d654-4e40-b1ea-276d830b09dc-kube-api-access-hprrs\") pod \"d5fc1805-d654-4e40-b1ea-276d830b09dc\" (UID: \"d5fc1805-d654-4e40-b1ea-276d830b09dc\") " Jul 24 17:29:24.584370 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:24.584249 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5fc1805-d654-4e40-b1ea-276d830b09dc-webhook-cert\") pod \"d5fc1805-d654-4e40-b1ea-276d830b09dc\" (UID: \"d5fc1805-d654-4e40-b1ea-276d830b09dc\") " Jul 24 17:29:24.586605 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:24.586577 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fc1805-d654-4e40-b1ea-276d830b09dc-kube-api-access-hprrs" (OuterVolumeSpecName: "kube-api-access-hprrs") pod "d5fc1805-d654-4e40-b1ea-276d830b09dc" (UID: "d5fc1805-d654-4e40-b1ea-276d830b09dc"). InnerVolumeSpecName "kube-api-access-hprrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:29:24.586723 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:24.586612 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5fc1805-d654-4e40-b1ea-276d830b09dc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d5fc1805-d654-4e40-b1ea-276d830b09dc" (UID: "d5fc1805-d654-4e40-b1ea-276d830b09dc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:29:24.685233 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:24.685139 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5fc1805-d654-4e40-b1ea-276d830b09dc-webhook-cert\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:29:24.685233 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:24.685172 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hprrs\" (UniqueName: \"kubernetes.io/projected/d5fc1805-d654-4e40-b1ea-276d830b09dc-kube-api-access-hprrs\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:29:25.143501 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:25.143466 2559 generic.go:356] "Generic (PLEG): container finished" podID="d5fc1805-d654-4e40-b1ea-276d830b09dc" containerID="b7aba81f45b2cf22270cda09bbf7369049fa5188cfb1da09707f83733caf137f" exitCode=0 Jul 24 17:29:25.143897 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:25.143549 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56dc868d8b-zw7k7" Jul 24 17:29:25.143897 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:25.143554 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56dc868d8b-zw7k7" event={"ID":"d5fc1805-d654-4e40-b1ea-276d830b09dc","Type":"ContainerDied","Data":"b7aba81f45b2cf22270cda09bbf7369049fa5188cfb1da09707f83733caf137f"} Jul 24 17:29:25.143897 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:25.143637 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56dc868d8b-zw7k7" event={"ID":"d5fc1805-d654-4e40-b1ea-276d830b09dc","Type":"ContainerDied","Data":"15f8c588f393f443a0af07000abee4e8477b5c0b347a0c085e4a62f25e43b795"} Jul 24 17:29:25.143897 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:25.143660 2559 scope.go:117] "RemoveContainer" containerID="b7aba81f45b2cf22270cda09bbf7369049fa5188cfb1da09707f83733caf137f" Jul 24 17:29:25.151605 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:25.151587 2559 scope.go:117] "RemoveContainer" containerID="b7aba81f45b2cf22270cda09bbf7369049fa5188cfb1da09707f83733caf137f" Jul 24 17:29:25.151890 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:29:25.151873 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7aba81f45b2cf22270cda09bbf7369049fa5188cfb1da09707f83733caf137f\": container with ID starting with b7aba81f45b2cf22270cda09bbf7369049fa5188cfb1da09707f83733caf137f not found: ID does not exist" containerID="b7aba81f45b2cf22270cda09bbf7369049fa5188cfb1da09707f83733caf137f" Jul 24 17:29:25.151963 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:25.151898 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7aba81f45b2cf22270cda09bbf7369049fa5188cfb1da09707f83733caf137f"} err="failed to get container status \"b7aba81f45b2cf22270cda09bbf7369049fa5188cfb1da09707f83733caf137f\": rpc error: code = NotFound desc = could not find container \"b7aba81f45b2cf22270cda09bbf7369049fa5188cfb1da09707f83733caf137f\": container with ID starting with b7aba81f45b2cf22270cda09bbf7369049fa5188cfb1da09707f83733caf137f not found: ID does not exist" Jul 24 17:29:25.160317 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:25.160298 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56dc868d8b-zw7k7"] Jul 24 17:29:25.164643 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:25.164626 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56dc868d8b-zw7k7"] Jul 24 17:29:26.942084 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:26.942048 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5fc1805-d654-4e40-b1ea-276d830b09dc" path="/var/lib/kubelet/pods/d5fc1805-d654-4e40-b1ea-276d830b09dc/volumes" Jul 24 17:29:45.490118 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.490080 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8b488659f-z6l8q"] Jul 24 17:29:45.490582 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.490361 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fc1805-d654-4e40-b1ea-276d830b09dc" containerName="manager" Jul 24 17:29:45.490582 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.490374 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fc1805-d654-4e40-b1ea-276d830b09dc" containerName="manager" Jul 24 17:29:45.490582 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.490428 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5fc1805-d654-4e40-b1ea-276d830b09dc" containerName="manager" Jul 24 17:29:45.508227 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.508196 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8b488659f-z6l8q"] Jul 24 17:29:45.508382 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.508301 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" Jul 24 17:29:45.511474 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.511446 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kxbbh\"" Jul 24 17:29:45.511624 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.511551 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 24 17:29:45.552150 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.552118 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d01e53-aae5-42eb-ba9b-bd275895eef1-openshift-service-ca\") pod \"maas-api-8b488659f-z6l8q\" (UID: \"c1d01e53-aae5-42eb-ba9b-bd275895eef1\") " pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" Jul 24 17:29:45.552306 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.552172 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d01e53-aae5-42eb-ba9b-bd275895eef1-maas-api-tls\") pod \"maas-api-8b488659f-z6l8q\" (UID: \"c1d01e53-aae5-42eb-ba9b-bd275895eef1\") " pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" Jul 24 17:29:45.552306 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.552194 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4b8t\" (UniqueName: \"kubernetes.io/projected/c1d01e53-aae5-42eb-ba9b-bd275895eef1-kube-api-access-t4b8t\") pod \"maas-api-8b488659f-z6l8q\" (UID: \"c1d01e53-aae5-42eb-ba9b-bd275895eef1\") " pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" Jul 24 17:29:45.653423 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.653386 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d01e53-aae5-42eb-ba9b-bd275895eef1-openshift-service-ca\") pod \"maas-api-8b488659f-z6l8q\" (UID: \"c1d01e53-aae5-42eb-ba9b-bd275895eef1\") " pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" Jul 24 17:29:45.653629 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.653452 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d01e53-aae5-42eb-ba9b-bd275895eef1-maas-api-tls\") pod \"maas-api-8b488659f-z6l8q\" (UID: \"c1d01e53-aae5-42eb-ba9b-bd275895eef1\") " pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" Jul 24 17:29:45.653629 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.653481 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4b8t\" (UniqueName: \"kubernetes.io/projected/c1d01e53-aae5-42eb-ba9b-bd275895eef1-kube-api-access-t4b8t\") pod \"maas-api-8b488659f-z6l8q\" (UID: \"c1d01e53-aae5-42eb-ba9b-bd275895eef1\") " pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" Jul 24 17:29:45.654223 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.654195 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d01e53-aae5-42eb-ba9b-bd275895eef1-openshift-service-ca\") pod \"maas-api-8b488659f-z6l8q\" (UID: \"c1d01e53-aae5-42eb-ba9b-bd275895eef1\") " pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" Jul 24 17:29:45.656196 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.656163 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d01e53-aae5-42eb-ba9b-bd275895eef1-maas-api-tls\") pod \"maas-api-8b488659f-z6l8q\" (UID: \"c1d01e53-aae5-42eb-ba9b-bd275895eef1\") " pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" Jul 24 17:29:45.661792 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.661767 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4b8t\" (UniqueName: \"kubernetes.io/projected/c1d01e53-aae5-42eb-ba9b-bd275895eef1-kube-api-access-t4b8t\") pod \"maas-api-8b488659f-z6l8q\" (UID: \"c1d01e53-aae5-42eb-ba9b-bd275895eef1\") " pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" Jul 24 17:29:45.819377 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.819342 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" Jul 24 17:29:45.988996 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:45.988959 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8b488659f-z6l8q"] Jul 24 17:29:45.993094 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:29:45.993069 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d01e53_aae5_42eb_ba9b_bd275895eef1.slice/crio-55a406e0b254a96da6de869edd19e4aee5eb555b4efd38bada190e6f66c055b7 WatchSource:0}: Error finding container 55a406e0b254a96da6de869edd19e4aee5eb555b4efd38bada190e6f66c055b7: Status 404 returned error can't find the container with id 55a406e0b254a96da6de869edd19e4aee5eb555b4efd38bada190e6f66c055b7 Jul 24 17:29:46.047665 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.047630 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk"] Jul 24 17:29:46.050280 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.050263 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk" Jul 24 17:29:46.054128 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.054105 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 17:29:46.055803 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.055780 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4402045-7fac-42d2-93be-8ccedb3d0db6-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-7mkdk\" (UID: \"d4402045-7fac-42d2-93be-8ccedb3d0db6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk" Jul 24 17:29:46.055923 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.055816 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvtcc\" (UniqueName: \"kubernetes.io/projected/d4402045-7fac-42d2-93be-8ccedb3d0db6-kube-api-access-cvtcc\") pod \"payload-pre-processing-687dbb6cf6-7mkdk\" (UID: \"d4402045-7fac-42d2-93be-8ccedb3d0db6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk" Jul 24 17:29:46.077117 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.077047 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk"] Jul 24 17:29:46.156243 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.156208 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4402045-7fac-42d2-93be-8ccedb3d0db6-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-7mkdk\" (UID: \"d4402045-7fac-42d2-93be-8ccedb3d0db6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk" Jul 24 17:29:46.156243 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.156242 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtcc\" (UniqueName: \"kubernetes.io/projected/d4402045-7fac-42d2-93be-8ccedb3d0db6-kube-api-access-cvtcc\") pod \"payload-pre-processing-687dbb6cf6-7mkdk\" (UID: \"d4402045-7fac-42d2-93be-8ccedb3d0db6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk" Jul 24 17:29:46.156969 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.156948 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4402045-7fac-42d2-93be-8ccedb3d0db6-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-7mkdk\" (UID: \"d4402045-7fac-42d2-93be-8ccedb3d0db6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk" Jul 24 17:29:46.175059 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.175033 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtcc\" (UniqueName: \"kubernetes.io/projected/d4402045-7fac-42d2-93be-8ccedb3d0db6-kube-api-access-cvtcc\") pod \"payload-pre-processing-687dbb6cf6-7mkdk\" (UID: \"d4402045-7fac-42d2-93be-8ccedb3d0db6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk" Jul 24 17:29:46.221704 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.221670 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-gq4hn"] Jul 24 17:29:46.224050 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.224034 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-gq4hn" Jul 24 17:29:46.228138 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.228107 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" event={"ID":"c1d01e53-aae5-42eb-ba9b-bd275895eef1","Type":"ContainerStarted","Data":"55a406e0b254a96da6de869edd19e4aee5eb555b4efd38bada190e6f66c055b7"} Jul 24 17:29:46.229365 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.229347 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nw5kl\"" Jul 24 17:29:46.250978 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.250960 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-gq4hn"] Jul 24 17:29:46.256901 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.256876 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e972f6f7-21a4-4495-b8a5-430a74b986c0-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-gq4hn\" (UID: \"e972f6f7-21a4-4495-b8a5-430a74b986c0\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gq4hn" Jul 24 17:29:46.257007 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.256989 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvdtd\" (UniqueName: \"kubernetes.io/projected/e972f6f7-21a4-4495-b8a5-430a74b986c0-kube-api-access-kvdtd\") pod \"payload-processing-5cc6cd444b-gq4hn\" (UID: \"e972f6f7-21a4-4495-b8a5-430a74b986c0\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gq4hn" Jul 24 17:29:46.357964 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.357879 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e972f6f7-21a4-4495-b8a5-430a74b986c0-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-gq4hn\" (UID: \"e972f6f7-21a4-4495-b8a5-430a74b986c0\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gq4hn" Jul 24 17:29:46.357964 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.357953 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvdtd\" (UniqueName: \"kubernetes.io/projected/e972f6f7-21a4-4495-b8a5-430a74b986c0-kube-api-access-kvdtd\") pod \"payload-processing-5cc6cd444b-gq4hn\" (UID: \"e972f6f7-21a4-4495-b8a5-430a74b986c0\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gq4hn" Jul 24 17:29:46.358668 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.358647 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e972f6f7-21a4-4495-b8a5-430a74b986c0-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-gq4hn\" (UID: \"e972f6f7-21a4-4495-b8a5-430a74b986c0\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gq4hn" Jul 24 17:29:46.359861 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.359843 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk" Jul 24 17:29:46.376438 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.376413 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvdtd\" (UniqueName: \"kubernetes.io/projected/e972f6f7-21a4-4495-b8a5-430a74b986c0-kube-api-access-kvdtd\") pod \"payload-processing-5cc6cd444b-gq4hn\" (UID: \"e972f6f7-21a4-4495-b8a5-430a74b986c0\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gq4hn" Jul 24 17:29:46.491001 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.490973 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk"] Jul 24 17:29:46.492983 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:29:46.492956 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4402045_7fac_42d2_93be_8ccedb3d0db6.slice/crio-4934d74b610d57ed3317a1d817af39b9c65380699b14902d248b15dbec82e7ab WatchSource:0}: Error finding container 4934d74b610d57ed3317a1d817af39b9c65380699b14902d248b15dbec82e7ab: Status 404 returned error can't find the container with id 4934d74b610d57ed3317a1d817af39b9c65380699b14902d248b15dbec82e7ab Jul 24 17:29:46.534369 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.534339 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-gq4hn" Jul 24 17:29:46.669899 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:46.669879 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-gq4hn"] Jul 24 17:29:46.673563 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:29:46.672604 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode972f6f7_21a4_4495_b8a5_430a74b986c0.slice/crio-a86be1873d2d6c67424ca8e937a5d3eb37d9f3cb768fbb1a5ebdd8c01f3b633e WatchSource:0}: Error finding container a86be1873d2d6c67424ca8e937a5d3eb37d9f3cb768fbb1a5ebdd8c01f3b633e: Status 404 returned error can't find the container with id a86be1873d2d6c67424ca8e937a5d3eb37d9f3cb768fbb1a5ebdd8c01f3b633e Jul 24 17:29:47.238590 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:47.238498 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-gq4hn" event={"ID":"e972f6f7-21a4-4495-b8a5-430a74b986c0","Type":"ContainerStarted","Data":"a86be1873d2d6c67424ca8e937a5d3eb37d9f3cb768fbb1a5ebdd8c01f3b633e"} Jul 24 17:29:47.242655 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:47.242218 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk" event={"ID":"d4402045-7fac-42d2-93be-8ccedb3d0db6","Type":"ContainerStarted","Data":"4934d74b610d57ed3317a1d817af39b9c65380699b14902d248b15dbec82e7ab"} Jul 24 17:29:48.248110 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:48.248076 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" event={"ID":"c1d01e53-aae5-42eb-ba9b-bd275895eef1","Type":"ContainerStarted","Data":"aa067f3adef4cbed71241309c79ff6b7d0297c43f68df3b3d720d1583cc0a7df"} Jul 24 17:29:48.248535 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:48.248243 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" Jul 24 17:29:48.268500 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:48.268451 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" podStartSLOduration=1.1673145090000001 podStartE2EDuration="3.268434215s" totalImagesPullingTime="2.101119706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:29:48.267042526 +0000 UTC m=+585.918934001" watchObservedRunningTime="2026-07-24 17:29:48.268434215 +0000 UTC m=+585.920325688" Jul 24 17:29:50.257643 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:50.257606 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-gq4hn" event={"ID":"e972f6f7-21a4-4495-b8a5-430a74b986c0","Type":"ContainerStarted","Data":"be535888e235b9da5063919258bb44743be50d9c1d5e9d17771f283111c63246"} Jul 24 17:29:50.258100 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:50.257665 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-gq4hn" Jul 24 17:29:50.259033 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:50.259012 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk" event={"ID":"d4402045-7fac-42d2-93be-8ccedb3d0db6","Type":"ContainerStarted","Data":"6728992bbac91ab02bb332fe4a8e3e92d64a39f7c1f33866570f2ee674b6fe84"} Jul 24 17:29:50.259143 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:50.259129 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk" Jul 24 17:29:50.275450 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:50.275412 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-gq4hn" podStartSLOduration=1.7233941860000002 podStartE2EDuration="4.27539813s" totalImagesPullingTime="2.552003944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:29:50.273843931 +0000 UTC m=+587.925735403" watchObservedRunningTime="2026-07-24 17:29:50.27539813 +0000 UTC m=+587.927289602" Jul 24 17:29:50.291396 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:50.291353 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk" podStartSLOduration=2.56724038 podStartE2EDuration="5.291340443s" totalImagesPullingTime="2.724100063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:29:50.289461166 +0000 UTC m=+587.941352639" watchObservedRunningTime="2026-07-24 17:29:50.291340443 +0000 UTC m=+587.943231916" Jul 24 17:29:54.258431 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:29:54.258399 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8b488659f-z6l8q" Jul 24 17:30:00.139835 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:00.139799 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748570-5tbbb"] Jul 24 17:30:00.142708 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:00.142689 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748570-5tbbb" Jul 24 17:30:00.146500 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:00.146480 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dxd7g\"" Jul 24 17:30:00.151907 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:00.151886 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748570-5tbbb" Jul 24 17:30:00.153888 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:00.153856 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748570-5tbbb"] Jul 24 17:30:00.284427 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:00.284405 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748570-5tbbb"] Jul 24 17:30:00.286939 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:30:00.286904 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6383c10_4603_47bd_8e4d_9a64a651b153.slice/crio-ba60cf607ecfd24aaea589b601a1701a9da7a83edd0bd1e18a8ab52d41280378 WatchSource:0}: Error finding container ba60cf607ecfd24aaea589b601a1701a9da7a83edd0bd1e18a8ab52d41280378: Status 404 returned error can't find the container with id ba60cf607ecfd24aaea589b601a1701a9da7a83edd0bd1e18a8ab52d41280378 Jul 24 17:30:00.304490 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:00.304464 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748570-5tbbb" event={"ID":"c6383c10-4603-47bd-8e4d-9a64a651b153","Type":"ContainerStarted","Data":"ba60cf607ecfd24aaea589b601a1701a9da7a83edd0bd1e18a8ab52d41280378"} Jul 24 17:30:01.266502 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:01.266474 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-gq4hn" Jul 24 17:30:01.266853 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:01.266560 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk" Jul 24 17:30:03.319487 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:03.319449 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748570-5tbbb" event={"ID":"c6383c10-4603-47bd-8e4d-9a64a651b153","Type":"ContainerStarted","Data":"ad0d8cd039b95417260c1e8054b22f571824ffbb9a9c5f3b378e43626847509e"} Jul 24 17:30:03.337757 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:03.337703 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748570-5tbbb" podStartSLOduration=0.991868643 podStartE2EDuration="3.337687912s" totalImagesPullingTime="2.345819269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:30:03.336126115 +0000 UTC m=+600.988017599" watchObservedRunningTime="2026-07-24 17:30:03.337687912 +0000 UTC m=+600.989579386" Jul 24 17:30:04.324852 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:04.324819 2559 generic.go:356] "Generic (PLEG): container finished" podID="c6383c10-4603-47bd-8e4d-9a64a651b153" containerID="ad0d8cd039b95417260c1e8054b22f571824ffbb9a9c5f3b378e43626847509e" exitCode=0 Jul 24 17:30:04.325240 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:04.324862 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748570-5tbbb" event={"ID":"c6383c10-4603-47bd-8e4d-9a64a651b153","Type":"ContainerDied","Data":"ad0d8cd039b95417260c1e8054b22f571824ffbb9a9c5f3b378e43626847509e"} Jul 24 17:30:05.460313 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:05.460289 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748570-5tbbb" Jul 24 17:30:06.335041 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:06.335014 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748570-5tbbb" Jul 24 17:30:06.335228 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:06.335008 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748570-5tbbb" event={"ID":"c6383c10-4603-47bd-8e4d-9a64a651b153","Type":"ContainerDied","Data":"ba60cf607ecfd24aaea589b601a1701a9da7a83edd0bd1e18a8ab52d41280378"} Jul 24 17:30:06.335228 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:06.335180 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba60cf607ecfd24aaea589b601a1701a9da7a83edd0bd1e18a8ab52d41280378" Jul 24 17:30:11.026429 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.026390 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb"] Jul 24 17:30:11.026837 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.026749 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6383c10-4603-47bd-8e4d-9a64a651b153" containerName="cleanup" Jul 24 17:30:11.026837 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.026761 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6383c10-4603-47bd-8e4d-9a64a651b153" containerName="cleanup" Jul 24 17:30:11.026837 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.026806 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6383c10-4603-47bd-8e4d-9a64a651b153" containerName="cleanup" Jul 24 17:30:11.029796 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.029778 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.032418 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.032399 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 17:30:11.033486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.033456 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 17:30:11.033486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.033477 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8zm82\"" Jul 24 17:30:11.033718 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.033500 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 24 17:30:11.043852 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.043831 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb"] Jul 24 17:30:11.161228 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.161192 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69ae1a7b-7ac0-4852-8299-520ccb85381b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.161228 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.161228 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69ae1a7b-7ac0-4852-8299-520ccb85381b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.161408 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.161248 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69ae1a7b-7ac0-4852-8299-520ccb85381b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.161408 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.161339 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69ae1a7b-7ac0-4852-8299-520ccb85381b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.161408 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.161389 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69ae1a7b-7ac0-4852-8299-520ccb85381b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.161526 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.161432 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qfp2\" (UniqueName: \"kubernetes.io/projected/69ae1a7b-7ac0-4852-8299-520ccb85381b-kube-api-access-4qfp2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.262886 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.262856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69ae1a7b-7ac0-4852-8299-520ccb85381b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.262886 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.262890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69ae1a7b-7ac0-4852-8299-520ccb85381b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.263105 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.262907 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69ae1a7b-7ac0-4852-8299-520ccb85381b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.263105 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.262941 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69ae1a7b-7ac0-4852-8299-520ccb85381b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.263105 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.262970 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69ae1a7b-7ac0-4852-8299-520ccb85381b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.263105 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.262994 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qfp2\" (UniqueName: \"kubernetes.io/projected/69ae1a7b-7ac0-4852-8299-520ccb85381b-kube-api-access-4qfp2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.263300 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.263238 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69ae1a7b-7ac0-4852-8299-520ccb85381b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.263300 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.263292 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69ae1a7b-7ac0-4852-8299-520ccb85381b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.263397 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.263330 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69ae1a7b-7ac0-4852-8299-520ccb85381b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.265492 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.265471 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69ae1a7b-7ac0-4852-8299-520ccb85381b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.265613 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.265604 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69ae1a7b-7ac0-4852-8299-520ccb85381b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.272860 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.272837 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qfp2\" (UniqueName: \"kubernetes.io/projected/69ae1a7b-7ac0-4852-8299-520ccb85381b-kube-api-access-4qfp2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb\" (UID: \"69ae1a7b-7ac0-4852-8299-520ccb85381b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.339607 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.339493 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:11.476576 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:11.476537 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb"] Jul 24 17:30:11.479640 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:30:11.479613 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ae1a7b_7ac0_4852_8299_520ccb85381b.slice/crio-95b2420c0b25051524c5d2261b577e3abddbf6b4e955662309f07652508daa8d WatchSource:0}: Error finding container 95b2420c0b25051524c5d2261b577e3abddbf6b4e955662309f07652508daa8d: Status 404 returned error can't find the container with id 95b2420c0b25051524c5d2261b577e3abddbf6b4e955662309f07652508daa8d Jul 24 17:30:12.361680 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:12.361644 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" event={"ID":"69ae1a7b-7ac0-4852-8299-520ccb85381b","Type":"ContainerStarted","Data":"95b2420c0b25051524c5d2261b577e3abddbf6b4e955662309f07652508daa8d"} Jul 24 17:30:13.367979 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:13.367942 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" event={"ID":"69ae1a7b-7ac0-4852-8299-520ccb85381b","Type":"ContainerStarted","Data":"55fd5cf324d70fa1d55bdda599d85196f9eddaf3905ba6844cfca0513303d29d"} Jul 24 17:30:13.386746 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:13.386697 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" podStartSLOduration=1.15381415 podStartE2EDuration="2.386681773s" totalImagesPullingTime="1.232867623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:30:13.38495031 +0000 UTC m=+611.036841795" watchObservedRunningTime="2026-07-24 17:30:13.386681773 +0000 UTC m=+611.038573245" Jul 24 17:30:21.340053 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:21.340013 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:21.347243 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:21.340082 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:21.352866 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:21.352841 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:21.417422 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:21.417396 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb" Jul 24 17:30:43.718309 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.718277 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf"] Jul 24 17:30:43.725563 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.725535 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.728455 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.728424 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 24 17:30:43.731500 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.731467 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf"] Jul 24 17:30:43.732892 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.732631 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbfbx\" (UniqueName: \"kubernetes.io/projected/18a3ca9f-627d-4374-b1db-8658bfde5557-kube-api-access-jbfbx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.732960 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.732728 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a3ca9f-627d-4374-b1db-8658bfde5557-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.733006 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.732979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18a3ca9f-627d-4374-b1db-8658bfde5557-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.733085 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.733033 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18a3ca9f-627d-4374-b1db-8658bfde5557-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.733154 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.733127 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18a3ca9f-627d-4374-b1db-8658bfde5557-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.733207 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.733174 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18a3ca9f-627d-4374-b1db-8658bfde5557-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.833840 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.833791 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18a3ca9f-627d-4374-b1db-8658bfde5557-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.834038 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.833864 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18a3ca9f-627d-4374-b1db-8658bfde5557-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.834038 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.833909 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18a3ca9f-627d-4374-b1db-8658bfde5557-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.834038 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.833928 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18a3ca9f-627d-4374-b1db-8658bfde5557-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.834038 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.833960 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbfbx\" (UniqueName: \"kubernetes.io/projected/18a3ca9f-627d-4374-b1db-8658bfde5557-kube-api-access-jbfbx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.834038 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.833989 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a3ca9f-627d-4374-b1db-8658bfde5557-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.834341 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.834310 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18a3ca9f-627d-4374-b1db-8658bfde5557-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.834432 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.834322 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18a3ca9f-627d-4374-b1db-8658bfde5557-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.834432 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.834387 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18a3ca9f-627d-4374-b1db-8658bfde5557-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.836452 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.836430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18a3ca9f-627d-4374-b1db-8658bfde5557-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.836566 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.836539 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a3ca9f-627d-4374-b1db-8658bfde5557-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:43.843847 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:43.843828 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbfbx\" (UniqueName: \"kubernetes.io/projected/18a3ca9f-627d-4374-b1db-8658bfde5557-kube-api-access-jbfbx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf\" (UID: \"18a3ca9f-627d-4374-b1db-8658bfde5557\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:44.036028 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:44.035989 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:44.165344 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:44.165321 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf"] Jul 24 17:30:44.167854 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:30:44.167828 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a3ca9f_627d_4374_b1db_8658bfde5557.slice/crio-8f59bf975d0f8e3c7a9c31d52fffa0659749ef6c92a18c2baeb210404baf91a4 WatchSource:0}: Error finding container 8f59bf975d0f8e3c7a9c31d52fffa0659749ef6c92a18c2baeb210404baf91a4: Status 404 returned error can't find the container with id 8f59bf975d0f8e3c7a9c31d52fffa0659749ef6c92a18c2baeb210404baf91a4 Jul 24 17:30:44.169223 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:44.169207 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:30:44.502199 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:44.502161 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" event={"ID":"18a3ca9f-627d-4374-b1db-8658bfde5557","Type":"ContainerStarted","Data":"95757d4bed865a771626c42052657c09a211e50e38563a45f6e1a50f97f309db"} Jul 24 17:30:44.502373 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:44.502241 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" event={"ID":"18a3ca9f-627d-4374-b1db-8658bfde5557","Type":"ContainerStarted","Data":"8f59bf975d0f8e3c7a9c31d52fffa0659749ef6c92a18c2baeb210404baf91a4"} Jul 24 17:30:44.519849 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:44.519800 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" podStartSLOduration=1.337948232 podStartE2EDuration="1.519779219s" totalImagesPullingTime="181.830987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:30:44.519463648 +0000 UTC m=+642.171355122" watchObservedRunningTime="2026-07-24 17:30:44.519779219 +0000 UTC m=+642.171670689" Jul 24 17:30:54.036695 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.036658 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:54.037068 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.036732 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:54.050015 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.049990 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:54.556046 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.556014 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf" Jul 24 17:30:54.807246 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.807173 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls"] Jul 24 17:30:54.810409 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.810393 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.812935 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.812916 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 24 17:30:54.822696 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.822663 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.822844 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.822707 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tndb\" (UniqueName: \"kubernetes.io/projected/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-kube-api-access-8tndb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.822844 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.822819 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.822948 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.822870 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.822948 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.822912 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.822948 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.822931 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.823891 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.823867 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls"] Jul 24 17:30:54.923949 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.923911 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.923949 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.923951 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tndb\" (UniqueName: \"kubernetes.io/projected/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-kube-api-access-8tndb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.924186 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.924019 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.924186 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.924048 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.924186 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.924086 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.924186 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.924112 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.924395 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.924341 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.924548 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.924477 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.924683 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.924615 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.926379 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.926360 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.926785 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.926768 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:54.931682 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:54.931660 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tndb\" (UniqueName: \"kubernetes.io/projected/62a7c06f-37aa-42ed-9180-ed3f60ceebdd-kube-api-access-8tndb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w9kls\" (UID: \"62a7c06f-37aa-42ed-9180-ed3f60ceebdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:55.120545 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:55.120443 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:30:55.256154 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:55.256127 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls"] Jul 24 17:30:55.257062 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:30:55.257038 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62a7c06f_37aa_42ed_9180_ed3f60ceebdd.slice/crio-9bfc4c63be70c2f7ce27bbf605b4298498c3f6db37248dee26f7c2b3eb8c0986 WatchSource:0}: Error finding container 9bfc4c63be70c2f7ce27bbf605b4298498c3f6db37248dee26f7c2b3eb8c0986: Status 404 returned error can't find the container with id 9bfc4c63be70c2f7ce27bbf605b4298498c3f6db37248dee26f7c2b3eb8c0986 Jul 24 17:30:55.550536 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:55.550484 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" event={"ID":"62a7c06f-37aa-42ed-9180-ed3f60ceebdd","Type":"ContainerStarted","Data":"399c41274b233e8c4f9ce91393962411b5470168023ea51483dae833e803ade2"} Jul 24 17:30:55.550689 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:55.550561 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" event={"ID":"62a7c06f-37aa-42ed-9180-ed3f60ceebdd","Type":"ContainerStarted","Data":"9bfc4c63be70c2f7ce27bbf605b4298498c3f6db37248dee26f7c2b3eb8c0986"} Jul 24 17:30:55.569420 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:30:55.569266 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" podStartSLOduration=1.3808644239999999 podStartE2EDuration="1.569245513s" totalImagesPullingTime="188.381089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:30:55.568750892 +0000 UTC m=+653.220642364" watchObservedRunningTime="2026-07-24 17:30:55.569245513 +0000 UTC m=+653.221136985" Jul 24 17:31:05.120776 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:31:05.120747 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:31:05.121209 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:31:05.120790 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:31:05.133284 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:31:05.133254 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:31:05.604067 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:31:05.604040 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w9kls" Jul 24 17:32:32.418551 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.418501 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-2458k"] Jul 24 17:32:32.419084 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.418743 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-56bbcff6f4-2458k" podUID="89cc0d44-2d10-4f6a-8862-afd36d054376" containerName="manager" containerID="cri-o://db959458a2c948b75f18cfa12a999f04a3fffc610fc2878951f0f631422c4573" gracePeriod=30 Jul 24 17:32:32.663332 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.663307 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bbcff6f4-2458k" Jul 24 17:32:32.781259 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.781231 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89cc0d44-2d10-4f6a-8862-afd36d054376-webhook-cert\") pod \"89cc0d44-2d10-4f6a-8862-afd36d054376\" (UID: \"89cc0d44-2d10-4f6a-8862-afd36d054376\") " Jul 24 17:32:32.781431 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.781306 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkmjl\" (UniqueName: \"kubernetes.io/projected/89cc0d44-2d10-4f6a-8862-afd36d054376-kube-api-access-xkmjl\") pod \"89cc0d44-2d10-4f6a-8862-afd36d054376\" (UID: \"89cc0d44-2d10-4f6a-8862-afd36d054376\") " Jul 24 17:32:32.783578 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.783545 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cc0d44-2d10-4f6a-8862-afd36d054376-kube-api-access-xkmjl" (OuterVolumeSpecName: "kube-api-access-xkmjl") pod "89cc0d44-2d10-4f6a-8862-afd36d054376" (UID: "89cc0d44-2d10-4f6a-8862-afd36d054376"). InnerVolumeSpecName "kube-api-access-xkmjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:32:32.783578 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.783557 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89cc0d44-2d10-4f6a-8862-afd36d054376-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "89cc0d44-2d10-4f6a-8862-afd36d054376" (UID: "89cc0d44-2d10-4f6a-8862-afd36d054376"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:32:32.882260 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.882227 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkmjl\" (UniqueName: \"kubernetes.io/projected/89cc0d44-2d10-4f6a-8862-afd36d054376-kube-api-access-xkmjl\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:32:32.882260 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.882257 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89cc0d44-2d10-4f6a-8862-afd36d054376-webhook-cert\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:32:32.960976 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.960942 2559 generic.go:356] "Generic (PLEG): container finished" podID="89cc0d44-2d10-4f6a-8862-afd36d054376" containerID="db959458a2c948b75f18cfa12a999f04a3fffc610fc2878951f0f631422c4573" exitCode=0 Jul 24 17:32:32.961121 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.960981 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bbcff6f4-2458k" Jul 24 17:32:32.961121 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.961039 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bbcff6f4-2458k" event={"ID":"89cc0d44-2d10-4f6a-8862-afd36d054376","Type":"ContainerDied","Data":"db959458a2c948b75f18cfa12a999f04a3fffc610fc2878951f0f631422c4573"} Jul 24 17:32:32.961121 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.961086 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bbcff6f4-2458k" event={"ID":"89cc0d44-2d10-4f6a-8862-afd36d054376","Type":"ContainerDied","Data":"ce3939690cc01bb0d042b15ebed288f6df89ba0295edcb669390eb7790ae63cd"} Jul 24 17:32:32.961121 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.961109 2559 scope.go:117] "RemoveContainer" containerID="db959458a2c948b75f18cfa12a999f04a3fffc610fc2878951f0f631422c4573" Jul 24 17:32:32.969347 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.969331 2559 scope.go:117] "RemoveContainer" containerID="db959458a2c948b75f18cfa12a999f04a3fffc610fc2878951f0f631422c4573" Jul 24 17:32:32.969629 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:32:32.969609 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db959458a2c948b75f18cfa12a999f04a3fffc610fc2878951f0f631422c4573\": container with ID starting with db959458a2c948b75f18cfa12a999f04a3fffc610fc2878951f0f631422c4573 not found: ID does not exist" containerID="db959458a2c948b75f18cfa12a999f04a3fffc610fc2878951f0f631422c4573" Jul 24 17:32:32.969700 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.969640 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db959458a2c948b75f18cfa12a999f04a3fffc610fc2878951f0f631422c4573"} err="failed to get container status \"db959458a2c948b75f18cfa12a999f04a3fffc610fc2878951f0f631422c4573\": rpc error: code = NotFound desc = could not find container \"db959458a2c948b75f18cfa12a999f04a3fffc610fc2878951f0f631422c4573\": container with ID starting with db959458a2c948b75f18cfa12a999f04a3fffc610fc2878951f0f631422c4573 not found: ID does not exist" Jul 24 17:32:32.978754 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.978732 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-2458k"] Jul 24 17:32:32.983756 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:32.983738 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-2458k"] Jul 24 17:32:34.943493 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:32:34.943456 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89cc0d44-2d10-4f6a-8862-afd36d054376" path="/var/lib/kubelet/pods/89cc0d44-2d10-4f6a-8862-afd36d054376/volumes" Jul 24 17:33:33.758038 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:33.757959 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-zskwf"] Jul 24 17:33:33.758502 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:33.758311 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89cc0d44-2d10-4f6a-8862-afd36d054376" containerName="manager" Jul 24 17:33:33.758502 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:33.758324 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="89cc0d44-2d10-4f6a-8862-afd36d054376" containerName="manager" Jul 24 17:33:33.758502 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:33.758384 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="89cc0d44-2d10-4f6a-8862-afd36d054376" containerName="manager" Jul 24 17:33:33.761341 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:33.761326 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bbcff6f4-zskwf" Jul 24 17:33:33.765214 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:33.765193 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 17:33:33.765335 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:33.765234 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9hhf2\"" Jul 24 17:33:33.765335 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:33.765276 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 17:33:33.770542 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:33.770499 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-zskwf"] Jul 24 17:33:33.911428 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:33.911389 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr8lw\" (UniqueName: \"kubernetes.io/projected/f0c6ff57-6724-4608-93e5-36ae52707d6e-kube-api-access-fr8lw\") pod \"maas-controller-56bbcff6f4-zskwf\" (UID: \"f0c6ff57-6724-4608-93e5-36ae52707d6e\") " pod="opendatahub/maas-controller-56bbcff6f4-zskwf" Jul 24 17:33:33.911634 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:33.911468 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0c6ff57-6724-4608-93e5-36ae52707d6e-webhook-cert\") pod \"maas-controller-56bbcff6f4-zskwf\" (UID: \"f0c6ff57-6724-4608-93e5-36ae52707d6e\") " pod="opendatahub/maas-controller-56bbcff6f4-zskwf" Jul 24 17:33:34.012888 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:34.012850 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0c6ff57-6724-4608-93e5-36ae52707d6e-webhook-cert\") pod \"maas-controller-56bbcff6f4-zskwf\" (UID: \"f0c6ff57-6724-4608-93e5-36ae52707d6e\") " pod="opendatahub/maas-controller-56bbcff6f4-zskwf" Jul 24 17:33:34.013059 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:34.012911 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr8lw\" (UniqueName: \"kubernetes.io/projected/f0c6ff57-6724-4608-93e5-36ae52707d6e-kube-api-access-fr8lw\") pod \"maas-controller-56bbcff6f4-zskwf\" (UID: \"f0c6ff57-6724-4608-93e5-36ae52707d6e\") " pod="opendatahub/maas-controller-56bbcff6f4-zskwf" Jul 24 17:33:34.015468 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:34.015444 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0c6ff57-6724-4608-93e5-36ae52707d6e-webhook-cert\") pod \"maas-controller-56bbcff6f4-zskwf\" (UID: \"f0c6ff57-6724-4608-93e5-36ae52707d6e\") " pod="opendatahub/maas-controller-56bbcff6f4-zskwf" Jul 24 17:33:34.023005 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:34.022978 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr8lw\" (UniqueName: \"kubernetes.io/projected/f0c6ff57-6724-4608-93e5-36ae52707d6e-kube-api-access-fr8lw\") pod \"maas-controller-56bbcff6f4-zskwf\" (UID: \"f0c6ff57-6724-4608-93e5-36ae52707d6e\") " pod="opendatahub/maas-controller-56bbcff6f4-zskwf" Jul 24 17:33:34.071331 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:34.071294 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bbcff6f4-zskwf" Jul 24 17:33:34.203312 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:34.203277 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-zskwf"] Jul 24 17:33:34.206225 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:33:34.206194 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c6ff57_6724_4608_93e5_36ae52707d6e.slice/crio-b024f54b1f84246daa992904a479aa19589d3c28b0ea5e6757abcef2895bb7b5 WatchSource:0}: Error finding container b024f54b1f84246daa992904a479aa19589d3c28b0ea5e6757abcef2895bb7b5: Status 404 returned error can't find the container with id b024f54b1f84246daa992904a479aa19589d3c28b0ea5e6757abcef2895bb7b5 Jul 24 17:33:35.215800 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:35.215763 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bbcff6f4-zskwf" event={"ID":"f0c6ff57-6724-4608-93e5-36ae52707d6e","Type":"ContainerStarted","Data":"b8081d8886909b77f4a7a4d8c5e26ed20b49f28f50aa5d27f8040784fd56d49b"} Jul 24 17:33:35.216190 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:35.215815 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bbcff6f4-zskwf" event={"ID":"f0c6ff57-6724-4608-93e5-36ae52707d6e","Type":"ContainerStarted","Data":"b024f54b1f84246daa992904a479aa19589d3c28b0ea5e6757abcef2895bb7b5"} Jul 24 17:33:35.216190 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:35.215895 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56bbcff6f4-zskwf" Jul 24 17:33:35.235628 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:35.235583 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56bbcff6f4-zskwf" podStartSLOduration=1.795871477 podStartE2EDuration="2.235567551s" totalImagesPullingTime="439.696074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:33:35.233131412 +0000 UTC m=+812.885022886" watchObservedRunningTime="2026-07-24 17:33:35.235567551 +0000 UTC m=+812.887459024" Jul 24 17:33:46.222723 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:46.222685 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56bbcff6f4-zskwf" Jul 24 17:33:50.832997 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:50.832957 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-zskwf"] Jul 24 17:33:50.833432 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:50.833158 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-56bbcff6f4-zskwf" podUID="f0c6ff57-6724-4608-93e5-36ae52707d6e" containerName="manager" containerID="cri-o://b8081d8886909b77f4a7a4d8c5e26ed20b49f28f50aa5d27f8040784fd56d49b" gracePeriod=30 Jul 24 17:33:51.081603 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.081577 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bbcff6f4-zskwf" Jul 24 17:33:51.163373 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.163292 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0c6ff57-6724-4608-93e5-36ae52707d6e-webhook-cert\") pod \"f0c6ff57-6724-4608-93e5-36ae52707d6e\" (UID: \"f0c6ff57-6724-4608-93e5-36ae52707d6e\") " Jul 24 17:33:51.163373 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.163369 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr8lw\" (UniqueName: \"kubernetes.io/projected/f0c6ff57-6724-4608-93e5-36ae52707d6e-kube-api-access-fr8lw\") pod \"f0c6ff57-6724-4608-93e5-36ae52707d6e\" (UID: \"f0c6ff57-6724-4608-93e5-36ae52707d6e\") " Jul 24 17:33:51.165355 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.165321 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0c6ff57-6724-4608-93e5-36ae52707d6e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f0c6ff57-6724-4608-93e5-36ae52707d6e" (UID: "f0c6ff57-6724-4608-93e5-36ae52707d6e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:33:51.165464 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.165371 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c6ff57-6724-4608-93e5-36ae52707d6e-kube-api-access-fr8lw" (OuterVolumeSpecName: "kube-api-access-fr8lw") pod "f0c6ff57-6724-4608-93e5-36ae52707d6e" (UID: "f0c6ff57-6724-4608-93e5-36ae52707d6e"). InnerVolumeSpecName "kube-api-access-fr8lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:33:51.263836 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.263807 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr8lw\" (UniqueName: \"kubernetes.io/projected/f0c6ff57-6724-4608-93e5-36ae52707d6e-kube-api-access-fr8lw\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:33:51.264000 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.263850 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0c6ff57-6724-4608-93e5-36ae52707d6e-webhook-cert\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:33:51.287935 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.287905 2559 generic.go:356] "Generic (PLEG): container finished" podID="f0c6ff57-6724-4608-93e5-36ae52707d6e" containerID="b8081d8886909b77f4a7a4d8c5e26ed20b49f28f50aa5d27f8040784fd56d49b" exitCode=0 Jul 24 17:33:51.288093 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.287951 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bbcff6f4-zskwf" Jul 24 17:33:51.288093 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.287956 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bbcff6f4-zskwf" event={"ID":"f0c6ff57-6724-4608-93e5-36ae52707d6e","Type":"ContainerDied","Data":"b8081d8886909b77f4a7a4d8c5e26ed20b49f28f50aa5d27f8040784fd56d49b"} Jul 24 17:33:51.288093 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.287997 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bbcff6f4-zskwf" event={"ID":"f0c6ff57-6724-4608-93e5-36ae52707d6e","Type":"ContainerDied","Data":"b024f54b1f84246daa992904a479aa19589d3c28b0ea5e6757abcef2895bb7b5"} Jul 24 17:33:51.288093 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.288018 2559 scope.go:117] "RemoveContainer" containerID="b8081d8886909b77f4a7a4d8c5e26ed20b49f28f50aa5d27f8040784fd56d49b" Jul 24 17:33:51.297129 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.297108 2559 scope.go:117] "RemoveContainer" containerID="b8081d8886909b77f4a7a4d8c5e26ed20b49f28f50aa5d27f8040784fd56d49b" Jul 24 17:33:51.297376 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:33:51.297356 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8081d8886909b77f4a7a4d8c5e26ed20b49f28f50aa5d27f8040784fd56d49b\": container with ID starting with b8081d8886909b77f4a7a4d8c5e26ed20b49f28f50aa5d27f8040784fd56d49b not found: ID does not exist" containerID="b8081d8886909b77f4a7a4d8c5e26ed20b49f28f50aa5d27f8040784fd56d49b" Jul 24 17:33:51.297427 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.297385 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8081d8886909b77f4a7a4d8c5e26ed20b49f28f50aa5d27f8040784fd56d49b"} err="failed to get container status \"b8081d8886909b77f4a7a4d8c5e26ed20b49f28f50aa5d27f8040784fd56d49b\": rpc error: code = NotFound desc = could not find container \"b8081d8886909b77f4a7a4d8c5e26ed20b49f28f50aa5d27f8040784fd56d49b\": container with ID starting with b8081d8886909b77f4a7a4d8c5e26ed20b49f28f50aa5d27f8040784fd56d49b not found: ID does not exist" Jul 24 17:33:51.310450 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.310429 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-zskwf"] Jul 24 17:33:51.314475 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:51.314456 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-zskwf"] Jul 24 17:33:52.093804 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.093774 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-x6jkx"] Jul 24 17:33:52.094193 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.094104 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0c6ff57-6724-4608-93e5-36ae52707d6e" containerName="manager" Jul 24 17:33:52.094193 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.094114 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0c6ff57-6724-4608-93e5-36ae52707d6e" containerName="manager" Jul 24 17:33:52.094193 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.094162 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0c6ff57-6724-4608-93e5-36ae52707d6e" containerName="manager" Jul 24 17:33:52.098385 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.098369 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bbcff6f4-x6jkx" Jul 24 17:33:52.103074 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.103053 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9hhf2\"" Jul 24 17:33:52.103201 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.103054 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 17:33:52.103201 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.103079 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 17:33:52.110011 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.109989 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-x6jkx"] Jul 24 17:33:52.170140 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.170111 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxfmq\" (UniqueName: \"kubernetes.io/projected/0aa8b217-bab8-4bee-9a0d-746adb489663-kube-api-access-zxfmq\") pod \"maas-controller-56bbcff6f4-x6jkx\" (UID: \"0aa8b217-bab8-4bee-9a0d-746adb489663\") " pod="opendatahub/maas-controller-56bbcff6f4-x6jkx" Jul 24 17:33:52.170305 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.170158 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0aa8b217-bab8-4bee-9a0d-746adb489663-webhook-cert\") pod \"maas-controller-56bbcff6f4-x6jkx\" (UID: \"0aa8b217-bab8-4bee-9a0d-746adb489663\") " pod="opendatahub/maas-controller-56bbcff6f4-x6jkx" Jul 24 17:33:52.270897 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.270867 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxfmq\" (UniqueName: \"kubernetes.io/projected/0aa8b217-bab8-4bee-9a0d-746adb489663-kube-api-access-zxfmq\") pod \"maas-controller-56bbcff6f4-x6jkx\" (UID: \"0aa8b217-bab8-4bee-9a0d-746adb489663\") " pod="opendatahub/maas-controller-56bbcff6f4-x6jkx" Jul 24 17:33:52.271078 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.270922 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0aa8b217-bab8-4bee-9a0d-746adb489663-webhook-cert\") pod \"maas-controller-56bbcff6f4-x6jkx\" (UID: \"0aa8b217-bab8-4bee-9a0d-746adb489663\") " pod="opendatahub/maas-controller-56bbcff6f4-x6jkx" Jul 24 17:33:52.273397 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.273374 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0aa8b217-bab8-4bee-9a0d-746adb489663-webhook-cert\") pod \"maas-controller-56bbcff6f4-x6jkx\" (UID: \"0aa8b217-bab8-4bee-9a0d-746adb489663\") " pod="opendatahub/maas-controller-56bbcff6f4-x6jkx" Jul 24 17:33:52.280439 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.280403 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxfmq\" (UniqueName: \"kubernetes.io/projected/0aa8b217-bab8-4bee-9a0d-746adb489663-kube-api-access-zxfmq\") pod \"maas-controller-56bbcff6f4-x6jkx\" (UID: \"0aa8b217-bab8-4bee-9a0d-746adb489663\") " pod="opendatahub/maas-controller-56bbcff6f4-x6jkx" Jul 24 17:33:52.408871 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.408772 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bbcff6f4-x6jkx" Jul 24 17:33:52.545902 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.545878 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56bbcff6f4-x6jkx"] Jul 24 17:33:52.547738 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:33:52.547708 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa8b217_bab8_4bee_9a0d_746adb489663.slice/crio-c876816697c567743f158ebb787258057c6a43258c8f07830c59d487758cbc4f WatchSource:0}: Error finding container c876816697c567743f158ebb787258057c6a43258c8f07830c59d487758cbc4f: Status 404 returned error can't find the container with id c876816697c567743f158ebb787258057c6a43258c8f07830c59d487758cbc4f Jul 24 17:33:52.942475 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:52.942448 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0c6ff57-6724-4608-93e5-36ae52707d6e" path="/var/lib/kubelet/pods/f0c6ff57-6724-4608-93e5-36ae52707d6e/volumes" Jul 24 17:33:53.298053 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:53.298019 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bbcff6f4-x6jkx" event={"ID":"0aa8b217-bab8-4bee-9a0d-746adb489663","Type":"ContainerStarted","Data":"2bc34ed1b1ab45cd7e1e70b5a813c2d812bde8f2cc79321c0e6cfb4ac73b5023"} Jul 24 17:33:53.298475 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:53.298075 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bbcff6f4-x6jkx" event={"ID":"0aa8b217-bab8-4bee-9a0d-746adb489663","Type":"ContainerStarted","Data":"c876816697c567743f158ebb787258057c6a43258c8f07830c59d487758cbc4f"} Jul 24 17:33:53.298475 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:53.298114 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56bbcff6f4-x6jkx" Jul 24 17:33:53.319015 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:33:53.318960 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56bbcff6f4-x6jkx" podStartSLOduration=0.966768354 podStartE2EDuration="1.318938413s" totalImagesPullingTime="352.170059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:33:53.317342186 +0000 UTC m=+830.969233658" watchObservedRunningTime="2026-07-24 17:33:53.318938413 +0000 UTC m=+830.970829885" Jul 24 17:34:04.304721 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:34:04.304689 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56bbcff6f4-x6jkx" Jul 24 17:42:20.527371 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.527340 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg"] Jul 24 17:42:20.530800 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.530783 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.533335 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.533316 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2a1163f6-kserve-self-signed-certs\"" Jul 24 17:42:20.546856 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.546834 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg"] Jul 24 17:42:20.616161 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.616119 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5mcm\" (UniqueName: \"kubernetes.io/projected/93d1acfc-df3d-4905-9bd4-082911896605-kube-api-access-l5mcm\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.616344 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.616174 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-home\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.616344 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.616199 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-model-cache\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.616344 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.616268 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-dshm\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.616344 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.616322 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-tmp-dir\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.616344 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.616342 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93d1acfc-df3d-4905-9bd4-082911896605-tls-certs\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.717823 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.717783 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-home\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.717823 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.717822 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-model-cache\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.718045 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.717855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-dshm\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.718045 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.717892 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-tmp-dir\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.718045 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.717907 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93d1acfc-df3d-4905-9bd4-082911896605-tls-certs\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.718045 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.717949 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5mcm\" (UniqueName: \"kubernetes.io/projected/93d1acfc-df3d-4905-9bd4-082911896605-kube-api-access-l5mcm\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.718256 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.718232 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-model-cache\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.718320 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.718261 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-tmp-dir\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.718320 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.718309 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-home\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.720405 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.720383 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-dshm\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.720575 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.720556 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93d1acfc-df3d-4905-9bd4-082911896605-tls-certs\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.726126 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.726105 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5mcm\" (UniqueName: \"kubernetes.io/projected/93d1acfc-df3d-4905-9bd4-082911896605-kube-api-access-l5mcm\") pod \"e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.840218 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.840130 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:20.979762 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:20.979726 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg"] Jul 24 17:42:21.458382 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:21.458346 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" event={"ID":"93d1acfc-df3d-4905-9bd4-082911896605","Type":"ContainerStarted","Data":"4cbbc93106bff197e1cb304db116cd1cfd906279ac789d9b843ad00e2dc61790"} Jul 24 17:42:21.458579 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:21.458392 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" event={"ID":"93d1acfc-df3d-4905-9bd4-082911896605","Type":"ContainerStarted","Data":"2532278a3e375560f7f4c56906c16286326108b4d3facdf09ba312d4e23d57c3"} Jul 24 17:42:21.477272 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:21.477222 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" podStartSLOduration=1.4772074800000001 podStartE2EDuration="1.47720748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:42:21.47515537 +0000 UTC m=+1339.127046856" watchObservedRunningTime="2026-07-24 17:42:21.47720748 +0000 UTC m=+1339.129098953" Jul 24 17:42:30.841105 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:30.840997 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:30.841105 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:30.841063 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:30.853840 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:30.853816 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:31.513397 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:31.513371 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:42:53.003077 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:42:53.003042 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg"] Jul 24 17:43:08.022814 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.022772 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" podUID="93d1acfc-df3d-4905-9bd4-082911896605" containerName="main" containerID="cri-o://4cbbc93106bff197e1cb304db116cd1cfd906279ac789d9b843ad00e2dc61790" gracePeriod=45 Jul 24 17:43:08.274928 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.274866 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:43:08.323602 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.323575 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-dshm\") pod \"93d1acfc-df3d-4905-9bd4-082911896605\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " Jul 24 17:43:08.323602 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.323605 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93d1acfc-df3d-4905-9bd4-082911896605-tls-certs\") pod \"93d1acfc-df3d-4905-9bd4-082911896605\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " Jul 24 17:43:08.323851 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.323624 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5mcm\" (UniqueName: \"kubernetes.io/projected/93d1acfc-df3d-4905-9bd4-082911896605-kube-api-access-l5mcm\") pod \"93d1acfc-df3d-4905-9bd4-082911896605\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " Jul 24 17:43:08.323851 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.323651 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-model-cache\") pod \"93d1acfc-df3d-4905-9bd4-082911896605\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " Jul 24 17:43:08.323851 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.323676 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-tmp-dir\") pod \"93d1acfc-df3d-4905-9bd4-082911896605\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " Jul 24 17:43:08.323851 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.323699 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-home\") pod \"93d1acfc-df3d-4905-9bd4-082911896605\" (UID: \"93d1acfc-df3d-4905-9bd4-082911896605\") " Jul 24 17:43:08.324048 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.323931 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-model-cache" (OuterVolumeSpecName: "model-cache") pod "93d1acfc-df3d-4905-9bd4-082911896605" (UID: "93d1acfc-df3d-4905-9bd4-082911896605"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:43:08.324048 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.323944 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "93d1acfc-df3d-4905-9bd4-082911896605" (UID: "93d1acfc-df3d-4905-9bd4-082911896605"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:43:08.324048 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.323985 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-home" (OuterVolumeSpecName: "home") pod "93d1acfc-df3d-4905-9bd4-082911896605" (UID: "93d1acfc-df3d-4905-9bd4-082911896605"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:43:08.326023 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.325994 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93d1acfc-df3d-4905-9bd4-082911896605-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93d1acfc-df3d-4905-9bd4-082911896605" (UID: "93d1acfc-df3d-4905-9bd4-082911896605"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:43:08.326023 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.326017 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d1acfc-df3d-4905-9bd4-082911896605-kube-api-access-l5mcm" (OuterVolumeSpecName: "kube-api-access-l5mcm") pod "93d1acfc-df3d-4905-9bd4-082911896605" (UID: "93d1acfc-df3d-4905-9bd4-082911896605"). InnerVolumeSpecName "kube-api-access-l5mcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:43:08.326162 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.326019 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-dshm" (OuterVolumeSpecName: "dshm") pod "93d1acfc-df3d-4905-9bd4-082911896605" (UID: "93d1acfc-df3d-4905-9bd4-082911896605"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:43:08.425305 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.425265 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5mcm\" (UniqueName: \"kubernetes.io/projected/93d1acfc-df3d-4905-9bd4-082911896605-kube-api-access-l5mcm\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:43:08.425305 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.425295 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-model-cache\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:43:08.425305 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.425305 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-tmp-dir\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:43:08.425305 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.425313 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-home\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:43:08.425589 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.425321 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93d1acfc-df3d-4905-9bd4-082911896605-dshm\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:43:08.425589 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.425331 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93d1acfc-df3d-4905-9bd4-082911896605-tls-certs\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:43:08.664990 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.664905 2559 generic.go:356] "Generic (PLEG): container finished" podID="93d1acfc-df3d-4905-9bd4-082911896605" containerID="4cbbc93106bff197e1cb304db116cd1cfd906279ac789d9b843ad00e2dc61790" exitCode=0 Jul 24 17:43:08.664990 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.664949 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" Jul 24 17:43:08.665156 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.664953 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" event={"ID":"93d1acfc-df3d-4905-9bd4-082911896605","Type":"ContainerDied","Data":"4cbbc93106bff197e1cb304db116cd1cfd906279ac789d9b843ad00e2dc61790"} Jul 24 17:43:08.665156 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.665074 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg" event={"ID":"93d1acfc-df3d-4905-9bd4-082911896605","Type":"ContainerDied","Data":"2532278a3e375560f7f4c56906c16286326108b4d3facdf09ba312d4e23d57c3"} Jul 24 17:43:08.665156 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.665094 2559 scope.go:117] "RemoveContainer" containerID="4cbbc93106bff197e1cb304db116cd1cfd906279ac789d9b843ad00e2dc61790" Jul 24 17:43:08.674381 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.674346 2559 scope.go:117] "RemoveContainer" containerID="4cbbc93106bff197e1cb304db116cd1cfd906279ac789d9b843ad00e2dc61790" Jul 24 17:43:08.674673 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:43:08.674655 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cbbc93106bff197e1cb304db116cd1cfd906279ac789d9b843ad00e2dc61790\": container with ID starting with 4cbbc93106bff197e1cb304db116cd1cfd906279ac789d9b843ad00e2dc61790 not found: ID does not exist" containerID="4cbbc93106bff197e1cb304db116cd1cfd906279ac789d9b843ad00e2dc61790" Jul 24 17:43:08.674729 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.674680 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cbbc93106bff197e1cb304db116cd1cfd906279ac789d9b843ad00e2dc61790"} err="failed to get container status \"4cbbc93106bff197e1cb304db116cd1cfd906279ac789d9b843ad00e2dc61790\": rpc error: code = NotFound desc = could not find container \"4cbbc93106bff197e1cb304db116cd1cfd906279ac789d9b843ad00e2dc61790\": container with ID starting with 4cbbc93106bff197e1cb304db116cd1cfd906279ac789d9b843ad00e2dc61790 not found: ID does not exist" Jul 24 17:43:08.687019 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.686989 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg"] Jul 24 17:43:08.692919 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.692886 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2a1163f6-kserve-9b99747-b5mbg"] Jul 24 17:43:08.943638 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:43:08.943557 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93d1acfc-df3d-4905-9bd4-082911896605" path="/var/lib/kubelet/pods/93d1acfc-df3d-4905-9bd4-082911896605/volumes" Jul 24 17:46:45.989549 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:45.989497 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj"] Jul 24 17:46:45.989988 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:45.989849 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93d1acfc-df3d-4905-9bd4-082911896605" containerName="main" Jul 24 17:46:45.989988 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:45.989861 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="93d1acfc-df3d-4905-9bd4-082911896605" containerName="main" Jul 24 17:46:45.989988 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:45.989908 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="93d1acfc-df3d-4905-9bd4-082911896605" containerName="main" Jul 24 17:46:45.993084 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:45.993068 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:46:45.996865 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:45.996849 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3102a353-serving-cert\"" Jul 24 17:46:46.006867 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.006844 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj"] Jul 24 17:46:46.095774 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.095742 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2eaa67-d874-409e-8fa9-341a863f6f47-openshift-service-ca\") pod \"maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj\" (UID: \"8a2eaa67-d874-409e-8fa9-341a863f6f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:46:46.095945 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.095783 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2eaa67-d874-409e-8fa9-341a863f6f47-maas-api-tls\") pod \"maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj\" (UID: \"8a2eaa67-d874-409e-8fa9-341a863f6f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:46:46.095945 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.095801 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t59bq\" (UniqueName: \"kubernetes.io/projected/8a2eaa67-d874-409e-8fa9-341a863f6f47-kube-api-access-t59bq\") pod \"maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj\" (UID: \"8a2eaa67-d874-409e-8fa9-341a863f6f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:46:46.196237 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.196201 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2eaa67-d874-409e-8fa9-341a863f6f47-maas-api-tls\") pod \"maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj\" (UID: \"8a2eaa67-d874-409e-8fa9-341a863f6f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:46:46.196237 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.196234 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t59bq\" (UniqueName: \"kubernetes.io/projected/8a2eaa67-d874-409e-8fa9-341a863f6f47-kube-api-access-t59bq\") pod \"maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj\" (UID: \"8a2eaa67-d874-409e-8fa9-341a863f6f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:46:46.196444 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.196300 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2eaa67-d874-409e-8fa9-341a863f6f47-openshift-service-ca\") pod \"maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj\" (UID: \"8a2eaa67-d874-409e-8fa9-341a863f6f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:46:46.196444 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:46:46.196355 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-serving-cert: secret "maas-api-e2e-ait-3102a353-serving-cert" not found Jul 24 17:46:46.196444 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:46:46.196424 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2eaa67-d874-409e-8fa9-341a863f6f47-maas-api-tls podName:8a2eaa67-d874-409e-8fa9-341a863f6f47 nodeName:}" failed. No retries permitted until 2026-07-24 17:46:46.69640589 +0000 UTC m=+1604.348297346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a2eaa67-d874-409e-8fa9-341a863f6f47-maas-api-tls") pod "maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" (UID: "8a2eaa67-d874-409e-8fa9-341a863f6f47") : secret "maas-api-e2e-ait-3102a353-serving-cert" not found Jul 24 17:46:46.196935 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.196918 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2eaa67-d874-409e-8fa9-341a863f6f47-openshift-service-ca\") pod \"maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj\" (UID: \"8a2eaa67-d874-409e-8fa9-341a863f6f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:46:46.205498 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.205470 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t59bq\" (UniqueName: \"kubernetes.io/projected/8a2eaa67-d874-409e-8fa9-341a863f6f47-kube-api-access-t59bq\") pod \"maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj\" (UID: \"8a2eaa67-d874-409e-8fa9-341a863f6f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:46:46.699834 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.699793 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2eaa67-d874-409e-8fa9-341a863f6f47-maas-api-tls\") pod \"maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj\" (UID: \"8a2eaa67-d874-409e-8fa9-341a863f6f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:46:46.702463 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.702440 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2eaa67-d874-409e-8fa9-341a863f6f47-maas-api-tls\") pod \"maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj\" (UID: \"8a2eaa67-d874-409e-8fa9-341a863f6f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:46:46.813816 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.813780 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz"] Jul 24 17:46:46.817191 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.817174 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" Jul 24 17:46:46.819847 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.819825 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3102a353-dockercfg-q48dp\"" Jul 24 17:46:46.820152 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.820136 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3102a353\"" Jul 24 17:46:46.825682 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.825660 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz"] Jul 24 17:46:46.830801 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.830780 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc"] Jul 24 17:46:46.833953 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.833938 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" Jul 24 17:46:46.845250 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.845230 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc"] Jul 24 17:46:46.901658 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.901620 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjtb4\" (UniqueName: \"kubernetes.io/projected/3d50242c-c65d-48b6-803a-6f41451bdbf6-kube-api-access-kjtb4\") pod \"payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc\" (UID: \"3d50242c-c65d-48b6-803a-6f41451bdbf6\") " pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" Jul 24 17:46:46.901819 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.901673 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f5ff0e-9d73-434a-b896-9df9133b2516-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz\" (UID: \"b6f5ff0e-9d73-434a-b896-9df9133b2516\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" Jul 24 17:46:46.901819 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.901695 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d50242c-c65d-48b6-803a-6f41451bdbf6-plugins-config-volume\") pod \"payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc\" (UID: \"3d50242c-c65d-48b6-803a-6f41451bdbf6\") " pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" Jul 24 17:46:46.901819 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.901741 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzg9f\" (UniqueName: \"kubernetes.io/projected/b6f5ff0e-9d73-434a-b896-9df9133b2516-kube-api-access-lzg9f\") pod \"payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz\" (UID: \"b6f5ff0e-9d73-434a-b896-9df9133b2516\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" Jul 24 17:46:46.903215 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:46.903197 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:46:47.003025 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.002829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f5ff0e-9d73-434a-b896-9df9133b2516-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz\" (UID: \"b6f5ff0e-9d73-434a-b896-9df9133b2516\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" Jul 24 17:46:47.003025 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.002877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d50242c-c65d-48b6-803a-6f41451bdbf6-plugins-config-volume\") pod \"payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc\" (UID: \"3d50242c-c65d-48b6-803a-6f41451bdbf6\") " pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" Jul 24 17:46:47.003506 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.003214 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzg9f\" (UniqueName: \"kubernetes.io/projected/b6f5ff0e-9d73-434a-b896-9df9133b2516-kube-api-access-lzg9f\") pod \"payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz\" (UID: \"b6f5ff0e-9d73-434a-b896-9df9133b2516\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" Jul 24 17:46:47.003506 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.003318 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjtb4\" (UniqueName: \"kubernetes.io/projected/3d50242c-c65d-48b6-803a-6f41451bdbf6-kube-api-access-kjtb4\") pod \"payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc\" (UID: \"3d50242c-c65d-48b6-803a-6f41451bdbf6\") " pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" Jul 24 17:46:47.003911 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.003865 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d50242c-c65d-48b6-803a-6f41451bdbf6-plugins-config-volume\") pod \"payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc\" (UID: \"3d50242c-c65d-48b6-803a-6f41451bdbf6\") " pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" Jul 24 17:46:47.004077 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.004054 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f5ff0e-9d73-434a-b896-9df9133b2516-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz\" (UID: \"b6f5ff0e-9d73-434a-b896-9df9133b2516\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" Jul 24 17:46:47.014616 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.014567 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjtb4\" (UniqueName: \"kubernetes.io/projected/3d50242c-c65d-48b6-803a-6f41451bdbf6-kube-api-access-kjtb4\") pod \"payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc\" (UID: \"3d50242c-c65d-48b6-803a-6f41451bdbf6\") " pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" Jul 24 17:46:47.016093 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.016072 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzg9f\" (UniqueName: \"kubernetes.io/projected/b6f5ff0e-9d73-434a-b896-9df9133b2516-kube-api-access-lzg9f\") pod \"payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz\" (UID: \"b6f5ff0e-9d73-434a-b896-9df9133b2516\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" Jul 24 17:46:47.053488 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.053466 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj"] Jul 24 17:46:47.056130 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:46:47.056105 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2eaa67_d874_409e_8fa9_341a863f6f47.slice/crio-c81640c7d8a110d8d359d5dc9c2e459d2dc3b38c3b761da9e261a65e151c5e8e WatchSource:0}: Error finding container c81640c7d8a110d8d359d5dc9c2e459d2dc3b38c3b761da9e261a65e151c5e8e: Status 404 returned error can't find the container with id c81640c7d8a110d8d359d5dc9c2e459d2dc3b38c3b761da9e261a65e151c5e8e Jul 24 17:46:47.057411 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.057397 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:46:47.128444 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.128412 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" Jul 24 17:46:47.144837 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.144809 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" Jul 24 17:46:47.291589 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.291567 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz"] Jul 24 17:46:47.293151 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:46:47.293127 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f5ff0e_9d73_434a_b896_9df9133b2516.slice/crio-bd7555fcd49f12825d5714726a1ae50c935ee81ad98ef4187ab62ff5177bfaac WatchSource:0}: Error finding container bd7555fcd49f12825d5714726a1ae50c935ee81ad98ef4187ab62ff5177bfaac: Status 404 returned error can't find the container with id bd7555fcd49f12825d5714726a1ae50c935ee81ad98ef4187ab62ff5177bfaac Jul 24 17:46:47.319337 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.319313 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc"] Jul 24 17:46:47.320581 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:46:47.320559 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d50242c_c65d_48b6_803a_6f41451bdbf6.slice/crio-acadb5784e63909c50e4d290b041697100db2103d81610a67fd0182e8639f8c0 WatchSource:0}: Error finding container acadb5784e63909c50e4d290b041697100db2103d81610a67fd0182e8639f8c0: Status 404 returned error can't find the container with id acadb5784e63909c50e4d290b041697100db2103d81610a67fd0182e8639f8c0 Jul 24 17:46:47.611876 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.611851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" event={"ID":"b6f5ff0e-9d73-434a-b896-9df9133b2516","Type":"ContainerStarted","Data":"e5aa4583c5fc9988b9dc95e5633612ed8fd952e246c90e191ff446c789df790f"} Jul 24 17:46:47.612115 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.611902 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" event={"ID":"b6f5ff0e-9d73-434a-b896-9df9133b2516","Type":"ContainerStarted","Data":"bd7555fcd49f12825d5714726a1ae50c935ee81ad98ef4187ab62ff5177bfaac"} Jul 24 17:46:47.612115 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.611941 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" Jul 24 17:46:47.613636 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.613612 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" event={"ID":"3d50242c-c65d-48b6-803a-6f41451bdbf6","Type":"ContainerStarted","Data":"cca5ddfb62aecd94696ddbb7f17c2e5fa802decb145fc247f3729bd6d54fd8b7"} Jul 24 17:46:47.613760 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.613658 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" event={"ID":"3d50242c-c65d-48b6-803a-6f41451bdbf6","Type":"ContainerStarted","Data":"acadb5784e63909c50e4d290b041697100db2103d81610a67fd0182e8639f8c0"} Jul 24 17:46:47.613760 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.613690 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" Jul 24 17:46:47.615244 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.615219 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" event={"ID":"8a2eaa67-d874-409e-8fa9-341a863f6f47","Type":"ContainerStarted","Data":"c81640c7d8a110d8d359d5dc9c2e459d2dc3b38c3b761da9e261a65e151c5e8e"} Jul 24 17:46:47.615327 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.615273 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:46:47.649503 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.649455 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" podStartSLOduration=1.649441478 podStartE2EDuration="1.649441478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:47.628966326 +0000 UTC m=+1605.280857800" watchObservedRunningTime="2026-07-24 17:46:47.649441478 +0000 UTC m=+1605.301332942" Jul 24 17:46:47.650347 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.650319 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" podStartSLOduration=1.650311691 podStartE2EDuration="1.650311691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:47.648673491 +0000 UTC m=+1605.300564964" watchObservedRunningTime="2026-07-24 17:46:47.650311691 +0000 UTC m=+1605.302203166" Jul 24 17:46:47.666446 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:47.666402 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" podStartSLOduration=2.175495557 podStartE2EDuration="2.666387403s" totalImagesPullingTime="490.891846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:47.665030862 +0000 UTC m=+1605.316922338" watchObservedRunningTime="2026-07-24 17:46:47.666387403 +0000 UTC m=+1605.318278876" Jul 24 17:46:48.620854 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:48.620811 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" event={"ID":"8a2eaa67-d874-409e-8fa9-341a863f6f47","Type":"ContainerStarted","Data":"ffa25da14ced68657c7fb196e8d9e144840de9469711882cc870116863173317"} Jul 24 17:46:54.632135 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:54.632109 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:46:58.622624 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:58.622592 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" Jul 24 17:46:58.623033 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:46:58.622832 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" Jul 24 17:52:03.189954 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.189916 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj"] Jul 24 17:52:03.190373 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.190180 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" podUID="8a2eaa67-d874-409e-8fa9-341a863f6f47" containerName="maas-api" containerID="cri-o://ffa25da14ced68657c7fb196e8d9e144840de9469711882cc870116863173317" gracePeriod=30 Jul 24 17:52:03.263093 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.263054 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc"] Jul 24 17:52:03.263330 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.263297 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" podUID="3d50242c-c65d-48b6-803a-6f41451bdbf6" containerName="payload-processing" containerID="cri-o://cca5ddfb62aecd94696ddbb7f17c2e5fa802decb145fc247f3729bd6d54fd8b7" gracePeriod=30 Jul 24 17:52:03.270111 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.269923 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz"] Jul 24 17:52:03.270252 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.270163 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" podUID="b6f5ff0e-9d73-434a-b896-9df9133b2516" containerName="payload-pre-processing" containerID="cri-o://e5aa4583c5fc9988b9dc95e5633612ed8fd952e246c90e191ff446c789df790f" gracePeriod=30 Jul 24 17:52:03.512686 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.512659 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:52:03.547814 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.547789 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" Jul 24 17:52:03.556100 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.556079 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" Jul 24 17:52:03.606540 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.606491 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2eaa67-d874-409e-8fa9-341a863f6f47-maas-api-tls\") pod \"8a2eaa67-d874-409e-8fa9-341a863f6f47\" (UID: \"8a2eaa67-d874-409e-8fa9-341a863f6f47\") " Jul 24 17:52:03.606540 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.606544 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjtb4\" (UniqueName: \"kubernetes.io/projected/3d50242c-c65d-48b6-803a-6f41451bdbf6-kube-api-access-kjtb4\") pod \"3d50242c-c65d-48b6-803a-6f41451bdbf6\" (UID: \"3d50242c-c65d-48b6-803a-6f41451bdbf6\") " Jul 24 17:52:03.606733 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.606597 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzg9f\" (UniqueName: \"kubernetes.io/projected/b6f5ff0e-9d73-434a-b896-9df9133b2516-kube-api-access-lzg9f\") pod \"b6f5ff0e-9d73-434a-b896-9df9133b2516\" (UID: \"b6f5ff0e-9d73-434a-b896-9df9133b2516\") " Jul 24 17:52:03.606733 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.606638 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t59bq\" (UniqueName: \"kubernetes.io/projected/8a2eaa67-d874-409e-8fa9-341a863f6f47-kube-api-access-t59bq\") pod \"8a2eaa67-d874-409e-8fa9-341a863f6f47\" (UID: \"8a2eaa67-d874-409e-8fa9-341a863f6f47\") " Jul 24 17:52:03.606733 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.606707 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2eaa67-d874-409e-8fa9-341a863f6f47-openshift-service-ca\") pod \"8a2eaa67-d874-409e-8fa9-341a863f6f47\" (UID: \"8a2eaa67-d874-409e-8fa9-341a863f6f47\") " Jul 24 17:52:03.606859 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.606787 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d50242c-c65d-48b6-803a-6f41451bdbf6-plugins-config-volume\") pod \"3d50242c-c65d-48b6-803a-6f41451bdbf6\" (UID: \"3d50242c-c65d-48b6-803a-6f41451bdbf6\") " Jul 24 17:52:03.606859 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.606821 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f5ff0e-9d73-434a-b896-9df9133b2516-plugins-config-volume\") pod \"b6f5ff0e-9d73-434a-b896-9df9133b2516\" (UID: \"b6f5ff0e-9d73-434a-b896-9df9133b2516\") " Jul 24 17:52:03.607335 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.607170 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2eaa67-d874-409e-8fa9-341a863f6f47-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a2eaa67-d874-409e-8fa9-341a863f6f47" (UID: "8a2eaa67-d874-409e-8fa9-341a863f6f47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:52:03.607459 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.607385 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d50242c-c65d-48b6-803a-6f41451bdbf6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d50242c-c65d-48b6-803a-6f41451bdbf6" (UID: "3d50242c-c65d-48b6-803a-6f41451bdbf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:52:03.607610 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.607584 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f5ff0e-9d73-434a-b896-9df9133b2516-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6f5ff0e-9d73-434a-b896-9df9133b2516" (UID: "b6f5ff0e-9d73-434a-b896-9df9133b2516"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:52:03.608961 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.608941 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d50242c-c65d-48b6-803a-6f41451bdbf6-kube-api-access-kjtb4" (OuterVolumeSpecName: "kube-api-access-kjtb4") pod "3d50242c-c65d-48b6-803a-6f41451bdbf6" (UID: "3d50242c-c65d-48b6-803a-6f41451bdbf6"). InnerVolumeSpecName "kube-api-access-kjtb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:52:03.609343 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.609310 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f5ff0e-9d73-434a-b896-9df9133b2516-kube-api-access-lzg9f" (OuterVolumeSpecName: "kube-api-access-lzg9f") pod "b6f5ff0e-9d73-434a-b896-9df9133b2516" (UID: "b6f5ff0e-9d73-434a-b896-9df9133b2516"). InnerVolumeSpecName "kube-api-access-lzg9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:52:03.609474 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.609383 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2eaa67-d874-409e-8fa9-341a863f6f47-kube-api-access-t59bq" (OuterVolumeSpecName: "kube-api-access-t59bq") pod "8a2eaa67-d874-409e-8fa9-341a863f6f47" (UID: "8a2eaa67-d874-409e-8fa9-341a863f6f47"). InnerVolumeSpecName "kube-api-access-t59bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:52:03.609474 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.609385 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a2eaa67-d874-409e-8fa9-341a863f6f47-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a2eaa67-d874-409e-8fa9-341a863f6f47" (UID: "8a2eaa67-d874-409e-8fa9-341a863f6f47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:52:03.707909 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.707873 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d50242c-c65d-48b6-803a-6f41451bdbf6-plugins-config-volume\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:52:03.707909 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.707901 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f5ff0e-9d73-434a-b896-9df9133b2516-plugins-config-volume\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:52:03.707909 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.707911 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2eaa67-d874-409e-8fa9-341a863f6f47-maas-api-tls\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:52:03.707909 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.707920 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjtb4\" (UniqueName: \"kubernetes.io/projected/3d50242c-c65d-48b6-803a-6f41451bdbf6-kube-api-access-kjtb4\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:52:03.708166 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.707928 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzg9f\" (UniqueName: \"kubernetes.io/projected/b6f5ff0e-9d73-434a-b896-9df9133b2516-kube-api-access-lzg9f\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:52:03.708166 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.707938 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t59bq\" (UniqueName: \"kubernetes.io/projected/8a2eaa67-d874-409e-8fa9-341a863f6f47-kube-api-access-t59bq\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:52:03.708166 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.707947 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2eaa67-d874-409e-8fa9-341a863f6f47-openshift-service-ca\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:52:03.995197 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.995164 2559 generic.go:356] "Generic (PLEG): container finished" podID="b6f5ff0e-9d73-434a-b896-9df9133b2516" containerID="e5aa4583c5fc9988b9dc95e5633612ed8fd952e246c90e191ff446c789df790f" exitCode=0 Jul 24 17:52:03.995384 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.995205 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" Jul 24 17:52:03.995384 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.995252 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" event={"ID":"b6f5ff0e-9d73-434a-b896-9df9133b2516","Type":"ContainerDied","Data":"e5aa4583c5fc9988b9dc95e5633612ed8fd952e246c90e191ff446c789df790f"} Jul 24 17:52:03.995384 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.995300 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz" event={"ID":"b6f5ff0e-9d73-434a-b896-9df9133b2516","Type":"ContainerDied","Data":"bd7555fcd49f12825d5714726a1ae50c935ee81ad98ef4187ab62ff5177bfaac"} Jul 24 17:52:03.995384 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.995325 2559 scope.go:117] "RemoveContainer" containerID="e5aa4583c5fc9988b9dc95e5633612ed8fd952e246c90e191ff446c789df790f" Jul 24 17:52:03.996665 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.996643 2559 generic.go:356] "Generic (PLEG): container finished" podID="3d50242c-c65d-48b6-803a-6f41451bdbf6" containerID="cca5ddfb62aecd94696ddbb7f17c2e5fa802decb145fc247f3729bd6d54fd8b7" exitCode=0 Jul 24 17:52:03.996778 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.996696 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" Jul 24 17:52:03.996778 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.996730 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" event={"ID":"3d50242c-c65d-48b6-803a-6f41451bdbf6","Type":"ContainerDied","Data":"cca5ddfb62aecd94696ddbb7f17c2e5fa802decb145fc247f3729bd6d54fd8b7"} Jul 24 17:52:03.996878 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.996781 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc" event={"ID":"3d50242c-c65d-48b6-803a-6f41451bdbf6","Type":"ContainerDied","Data":"acadb5784e63909c50e4d290b041697100db2103d81610a67fd0182e8639f8c0"} Jul 24 17:52:03.998253 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.998228 2559 generic.go:356] "Generic (PLEG): container finished" podID="8a2eaa67-d874-409e-8fa9-341a863f6f47" containerID="ffa25da14ced68657c7fb196e8d9e144840de9469711882cc870116863173317" exitCode=0 Jul 24 17:52:03.998360 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.998268 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" event={"ID":"8a2eaa67-d874-409e-8fa9-341a863f6f47","Type":"ContainerDied","Data":"ffa25da14ced68657c7fb196e8d9e144840de9469711882cc870116863173317"} Jul 24 17:52:03.998360 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.998286 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" Jul 24 17:52:03.998360 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:03.998307 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj" event={"ID":"8a2eaa67-d874-409e-8fa9-341a863f6f47","Type":"ContainerDied","Data":"c81640c7d8a110d8d359d5dc9c2e459d2dc3b38c3b761da9e261a65e151c5e8e"} Jul 24 17:52:04.004575 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.004552 2559 scope.go:117] "RemoveContainer" containerID="e5aa4583c5fc9988b9dc95e5633612ed8fd952e246c90e191ff446c789df790f" Jul 24 17:52:04.004812 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:52:04.004796 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5aa4583c5fc9988b9dc95e5633612ed8fd952e246c90e191ff446c789df790f\": container with ID starting with e5aa4583c5fc9988b9dc95e5633612ed8fd952e246c90e191ff446c789df790f not found: ID does not exist" containerID="e5aa4583c5fc9988b9dc95e5633612ed8fd952e246c90e191ff446c789df790f" Jul 24 17:52:04.004887 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.004819 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5aa4583c5fc9988b9dc95e5633612ed8fd952e246c90e191ff446c789df790f"} err="failed to get container status \"e5aa4583c5fc9988b9dc95e5633612ed8fd952e246c90e191ff446c789df790f\": rpc error: code = NotFound desc = could not find container \"e5aa4583c5fc9988b9dc95e5633612ed8fd952e246c90e191ff446c789df790f\": container with ID starting with e5aa4583c5fc9988b9dc95e5633612ed8fd952e246c90e191ff446c789df790f not found: ID does not exist" Jul 24 17:52:04.004887 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.004835 2559 scope.go:117] "RemoveContainer" containerID="cca5ddfb62aecd94696ddbb7f17c2e5fa802decb145fc247f3729bd6d54fd8b7" Jul 24 17:52:04.014315 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.014299 2559 scope.go:117] "RemoveContainer" containerID="cca5ddfb62aecd94696ddbb7f17c2e5fa802decb145fc247f3729bd6d54fd8b7" Jul 24 17:52:04.014575 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:52:04.014555 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cca5ddfb62aecd94696ddbb7f17c2e5fa802decb145fc247f3729bd6d54fd8b7\": container with ID starting with cca5ddfb62aecd94696ddbb7f17c2e5fa802decb145fc247f3729bd6d54fd8b7 not found: ID does not exist" containerID="cca5ddfb62aecd94696ddbb7f17c2e5fa802decb145fc247f3729bd6d54fd8b7" Jul 24 17:52:04.014644 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.014581 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cca5ddfb62aecd94696ddbb7f17c2e5fa802decb145fc247f3729bd6d54fd8b7"} err="failed to get container status \"cca5ddfb62aecd94696ddbb7f17c2e5fa802decb145fc247f3729bd6d54fd8b7\": rpc error: code = NotFound desc = could not find container \"cca5ddfb62aecd94696ddbb7f17c2e5fa802decb145fc247f3729bd6d54fd8b7\": container with ID starting with cca5ddfb62aecd94696ddbb7f17c2e5fa802decb145fc247f3729bd6d54fd8b7 not found: ID does not exist" Jul 24 17:52:04.014644 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.014596 2559 scope.go:117] "RemoveContainer" containerID="ffa25da14ced68657c7fb196e8d9e144840de9469711882cc870116863173317" Jul 24 17:52:04.022167 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.022146 2559 scope.go:117] "RemoveContainer" containerID="ffa25da14ced68657c7fb196e8d9e144840de9469711882cc870116863173317" Jul 24 17:52:04.022424 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:52:04.022403 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa25da14ced68657c7fb196e8d9e144840de9469711882cc870116863173317\": container with ID starting with ffa25da14ced68657c7fb196e8d9e144840de9469711882cc870116863173317 not found: ID does not exist" containerID="ffa25da14ced68657c7fb196e8d9e144840de9469711882cc870116863173317" Jul 24 17:52:04.022472 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.022431 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa25da14ced68657c7fb196e8d9e144840de9469711882cc870116863173317"} err="failed to get container status \"ffa25da14ced68657c7fb196e8d9e144840de9469711882cc870116863173317\": rpc error: code = NotFound desc = could not find container \"ffa25da14ced68657c7fb196e8d9e144840de9469711882cc870116863173317\": container with ID starting with ffa25da14ced68657c7fb196e8d9e144840de9469711882cc870116863173317 not found: ID does not exist" Jul 24 17:52:04.023748 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.023732 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc"] Jul 24 17:52:04.028278 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.028258 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc"] Jul 24 17:52:04.044138 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.044113 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj"] Jul 24 17:52:04.058003 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.057981 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3102a353-84cfcdbd4b-44xjj"] Jul 24 17:52:04.069827 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.069806 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz"] Jul 24 17:52:04.077804 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.077783 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz"] Jul 24 17:52:04.942777 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.942746 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d50242c-c65d-48b6-803a-6f41451bdbf6" path="/var/lib/kubelet/pods/3d50242c-c65d-48b6-803a-6f41451bdbf6/volumes" Jul 24 17:52:04.943142 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.943114 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a2eaa67-d874-409e-8fa9-341a863f6f47" path="/var/lib/kubelet/pods/8a2eaa67-d874-409e-8fa9-341a863f6f47/volumes" Jul 24 17:52:04.943410 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:04.943398 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6f5ff0e-9d73-434a-b896-9df9133b2516" path="/var/lib/kubelet/pods/b6f5ff0e-9d73-434a-b896-9df9133b2516/volumes" Jul 24 17:52:11.775189 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.775158 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w"] Jul 24 17:52:11.775593 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.775488 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f5ff0e-9d73-434a-b896-9df9133b2516" containerName="payload-pre-processing" Jul 24 17:52:11.775593 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.775499 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f5ff0e-9d73-434a-b896-9df9133b2516" containerName="payload-pre-processing" Jul 24 17:52:11.775593 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.775506 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d50242c-c65d-48b6-803a-6f41451bdbf6" containerName="payload-processing" Jul 24 17:52:11.775593 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.775525 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d50242c-c65d-48b6-803a-6f41451bdbf6" containerName="payload-processing" Jul 24 17:52:11.775593 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.775537 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a2eaa67-d874-409e-8fa9-341a863f6f47" containerName="maas-api" Jul 24 17:52:11.775593 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.775558 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a2eaa67-d874-409e-8fa9-341a863f6f47" containerName="maas-api" Jul 24 17:52:11.775790 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.775615 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f5ff0e-9d73-434a-b896-9df9133b2516" containerName="payload-pre-processing" Jul 24 17:52:11.775790 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.775623 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a2eaa67-d874-409e-8fa9-341a863f6f47" containerName="maas-api" Jul 24 17:52:11.775790 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.775629 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d50242c-c65d-48b6-803a-6f41451bdbf6" containerName="payload-processing" Jul 24 17:52:11.779972 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.779948 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:52:11.782398 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.782380 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-54f231e4-serving-cert\"" Jul 24 17:52:11.789040 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.789012 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w"] Jul 24 17:52:11.878658 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.878630 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-openshift-service-ca\") pod \"maas-api-e2e-migrate-54f231e4-845647678c-m6w5w\" (UID: \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:52:11.878820 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.878662 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnk47\" (UniqueName: \"kubernetes.io/projected/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-kube-api-access-wnk47\") pod \"maas-api-e2e-migrate-54f231e4-845647678c-m6w5w\" (UID: \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:52:11.878820 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.878682 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-maas-api-tls\") pod \"maas-api-e2e-migrate-54f231e4-845647678c-m6w5w\" (UID: \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:52:11.979674 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.979635 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-openshift-service-ca\") pod \"maas-api-e2e-migrate-54f231e4-845647678c-m6w5w\" (UID: \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:52:11.979674 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.979668 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnk47\" (UniqueName: \"kubernetes.io/projected/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-kube-api-access-wnk47\") pod \"maas-api-e2e-migrate-54f231e4-845647678c-m6w5w\" (UID: \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:52:11.979889 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.979692 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-maas-api-tls\") pod \"maas-api-e2e-migrate-54f231e4-845647678c-m6w5w\" (UID: \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:52:11.979889 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:52:11.979834 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-serving-cert: secret "maas-api-e2e-migrate-54f231e4-serving-cert" not found Jul 24 17:52:11.979970 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:52:11.979893 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-maas-api-tls podName:11f328eb-ce9e-44ec-b12d-41850a5e7d2f nodeName:}" failed. No retries permitted until 2026-07-24 17:52:12.479877642 +0000 UTC m=+1930.131769093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-maas-api-tls") pod "maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" (UID: "11f328eb-ce9e-44ec-b12d-41850a5e7d2f") : secret "maas-api-e2e-migrate-54f231e4-serving-cert" not found Jul 24 17:52:11.980318 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:11.980298 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-openshift-service-ca\") pod \"maas-api-e2e-migrate-54f231e4-845647678c-m6w5w\" (UID: \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:52:12.004925 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.004896 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnk47\" (UniqueName: \"kubernetes.io/projected/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-kube-api-access-wnk47\") pod \"maas-api-e2e-migrate-54f231e4-845647678c-m6w5w\" (UID: \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:52:12.483851 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.483822 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-maas-api-tls\") pod \"maas-api-e2e-migrate-54f231e4-845647678c-m6w5w\" (UID: \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:52:12.486289 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.486263 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-maas-api-tls\") pod \"maas-api-e2e-migrate-54f231e4-845647678c-m6w5w\" (UID: \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:52:12.583780 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.583753 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487"] Jul 24 17:52:12.587103 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.587086 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" Jul 24 17:52:12.589736 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.589714 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-54f231e4\"" Jul 24 17:52:12.589951 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.589936 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-54f231e4-dockercfg-c7qm5\"" Jul 24 17:52:12.601682 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.601660 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487"] Jul 24 17:52:12.639305 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.639279 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5"] Jul 24 17:52:12.642464 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.642450 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" Jul 24 17:52:12.655556 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.655535 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5"] Jul 24 17:52:12.685589 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.685553 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rzdj\" (UniqueName: \"kubernetes.io/projected/ace0e865-eb2f-4cae-9205-0c30017ba0a5-kube-api-access-4rzdj\") pod \"payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487\" (UID: \"ace0e865-eb2f-4cae-9205-0c30017ba0a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" Jul 24 17:52:12.685714 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.685610 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562692de-ff57-4f1d-b9b5-30f8e0a52bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5\" (UID: \"562692de-ff57-4f1d-b9b5-30f8e0a52bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" Jul 24 17:52:12.685714 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.685628 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9l4z\" (UniqueName: \"kubernetes.io/projected/562692de-ff57-4f1d-b9b5-30f8e0a52bd0-kube-api-access-v9l4z\") pod \"payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5\" (UID: \"562692de-ff57-4f1d-b9b5-30f8e0a52bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" Jul 24 17:52:12.685714 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.685653 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace0e865-eb2f-4cae-9205-0c30017ba0a5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487\" (UID: \"ace0e865-eb2f-4cae-9205-0c30017ba0a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" Jul 24 17:52:12.690493 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.690471 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:52:12.786442 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.786407 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rzdj\" (UniqueName: \"kubernetes.io/projected/ace0e865-eb2f-4cae-9205-0c30017ba0a5-kube-api-access-4rzdj\") pod \"payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487\" (UID: \"ace0e865-eb2f-4cae-9205-0c30017ba0a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" Jul 24 17:52:12.786814 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.786473 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562692de-ff57-4f1d-b9b5-30f8e0a52bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5\" (UID: \"562692de-ff57-4f1d-b9b5-30f8e0a52bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" Jul 24 17:52:12.786814 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.786507 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9l4z\" (UniqueName: \"kubernetes.io/projected/562692de-ff57-4f1d-b9b5-30f8e0a52bd0-kube-api-access-v9l4z\") pod \"payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5\" (UID: \"562692de-ff57-4f1d-b9b5-30f8e0a52bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" Jul 24 17:52:12.786814 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.786562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace0e865-eb2f-4cae-9205-0c30017ba0a5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487\" (UID: \"ace0e865-eb2f-4cae-9205-0c30017ba0a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" Jul 24 17:52:12.787422 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.787399 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562692de-ff57-4f1d-b9b5-30f8e0a52bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5\" (UID: \"562692de-ff57-4f1d-b9b5-30f8e0a52bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" Jul 24 17:52:12.787486 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.787450 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace0e865-eb2f-4cae-9205-0c30017ba0a5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487\" (UID: \"ace0e865-eb2f-4cae-9205-0c30017ba0a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" Jul 24 17:52:12.796234 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.796178 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9l4z\" (UniqueName: \"kubernetes.io/projected/562692de-ff57-4f1d-b9b5-30f8e0a52bd0-kube-api-access-v9l4z\") pod \"payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5\" (UID: \"562692de-ff57-4f1d-b9b5-30f8e0a52bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" Jul 24 17:52:12.800160 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.799963 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rzdj\" (UniqueName: \"kubernetes.io/projected/ace0e865-eb2f-4cae-9205-0c30017ba0a5-kube-api-access-4rzdj\") pod \"payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487\" (UID: \"ace0e865-eb2f-4cae-9205-0c30017ba0a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" Jul 24 17:52:12.830396 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.830371 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w"] Jul 24 17:52:12.832599 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:52:12.832574 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f328eb_ce9e_44ec_b12d_41850a5e7d2f.slice/crio-74fe6a5549ec39937ea7c1f167832bbf088fed9afdbe1b17df41a7be1f96304c WatchSource:0}: Error finding container 74fe6a5549ec39937ea7c1f167832bbf088fed9afdbe1b17df41a7be1f96304c: Status 404 returned error can't find the container with id 74fe6a5549ec39937ea7c1f167832bbf088fed9afdbe1b17df41a7be1f96304c Jul 24 17:52:12.834026 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.834009 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:52:12.896820 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.896789 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" Jul 24 17:52:12.952782 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:12.952571 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" Jul 24 17:52:13.044154 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:13.044121 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" event={"ID":"11f328eb-ce9e-44ec-b12d-41850a5e7d2f","Type":"ContainerStarted","Data":"74fe6a5549ec39937ea7c1f167832bbf088fed9afdbe1b17df41a7be1f96304c"} Jul 24 17:52:13.044682 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:13.044654 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487"] Jul 24 17:52:13.046420 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:52:13.046386 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace0e865_eb2f_4cae_9205_0c30017ba0a5.slice/crio-448cdc742254e129d836c09845f7515790bdaf8a2e25a7ee28331f2729b69c10 WatchSource:0}: Error finding container 448cdc742254e129d836c09845f7515790bdaf8a2e25a7ee28331f2729b69c10: Status 404 returned error can't find the container with id 448cdc742254e129d836c09845f7515790bdaf8a2e25a7ee28331f2729b69c10 Jul 24 17:52:13.108236 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:13.104658 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5"] Jul 24 17:52:14.050812 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:14.050777 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" event={"ID":"11f328eb-ce9e-44ec-b12d-41850a5e7d2f","Type":"ContainerStarted","Data":"c44ba5f0541dfb45d422e614ca9a7d015a5199332de33faefc5619e0e90afa5e"} Jul 24 17:52:14.051218 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:14.050903 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:52:14.052411 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:14.052388 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" event={"ID":"ace0e865-eb2f-4cae-9205-0c30017ba0a5","Type":"ContainerStarted","Data":"0f55799d1095c56dfbc01df6a2c9a969e6ae962053f5b5e78ea21dbbec870b1f"} Jul 24 17:52:14.052539 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:14.052428 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" event={"ID":"ace0e865-eb2f-4cae-9205-0c30017ba0a5","Type":"ContainerStarted","Data":"448cdc742254e129d836c09845f7515790bdaf8a2e25a7ee28331f2729b69c10"} Jul 24 17:52:14.052539 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:14.052458 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" Jul 24 17:52:14.053911 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:14.053884 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" event={"ID":"562692de-ff57-4f1d-b9b5-30f8e0a52bd0","Type":"ContainerStarted","Data":"b950e52f929f94a5754adf980fadba9adbe976ca1425d55e2f823a0f1923042a"} Jul 24 17:52:14.054006 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:14.053932 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" event={"ID":"562692de-ff57-4f1d-b9b5-30f8e0a52bd0","Type":"ContainerStarted","Data":"5026e6f4182e908e2fc9c9198ecbcc4ba1baa99920c944ed2cd7ff7f463d066e"} Jul 24 17:52:14.054006 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:14.053957 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" Jul 24 17:52:14.069601 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:14.069536 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" podStartSLOduration=2.544588678 podStartE2EDuration="3.069503208s" totalImagesPullingTime="524.91453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:52:14.067932862 +0000 UTC m=+1931.719824336" watchObservedRunningTime="2026-07-24 17:52:14.069503208 +0000 UTC m=+1931.721394681" Jul 24 17:52:14.086641 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:14.086605 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" podStartSLOduration=2.086590654 podStartE2EDuration="2.086590654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:52:14.08557486 +0000 UTC m=+1931.737466337" watchObservedRunningTime="2026-07-24 17:52:14.086590654 +0000 UTC m=+1931.738482126" Jul 24 17:52:14.101560 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:14.101411 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" podStartSLOduration=2.101395658 podStartE2EDuration="2.101395658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:52:14.101202366 +0000 UTC m=+1931.753093841" watchObservedRunningTime="2026-07-24 17:52:14.101395658 +0000 UTC m=+1931.753287133" Jul 24 17:52:20.063717 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:20.063687 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:52:25.060626 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:25.060593 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" Jul 24 17:52:25.061077 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:52:25.060667 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" Jul 24 17:55:30.770136 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:30.770093 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w"] Jul 24 17:55:30.770603 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:30.770298 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" podUID="11f328eb-ce9e-44ec-b12d-41850a5e7d2f" containerName="maas-api" containerID="cri-o://c44ba5f0541dfb45d422e614ca9a7d015a5199332de33faefc5619e0e90afa5e" gracePeriod=30 Jul 24 17:55:30.880916 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:30.880882 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487"] Jul 24 17:55:30.881224 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:30.881170 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" podUID="ace0e865-eb2f-4cae-9205-0c30017ba0a5" containerName="payload-processing" containerID="cri-o://0f55799d1095c56dfbc01df6a2c9a969e6ae962053f5b5e78ea21dbbec870b1f" gracePeriod=30 Jul 24 17:55:30.892141 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:30.892113 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5"] Jul 24 17:55:30.892376 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:30.892344 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" podUID="562692de-ff57-4f1d-b9b5-30f8e0a52bd0" containerName="payload-pre-processing" containerID="cri-o://b950e52f929f94a5754adf980fadba9adbe976ca1425d55e2f823a0f1923042a" gracePeriod=30 Jul 24 17:55:30.947983 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:30.947953 2559 generic.go:356] "Generic (PLEG): container finished" podID="11f328eb-ce9e-44ec-b12d-41850a5e7d2f" containerID="c44ba5f0541dfb45d422e614ca9a7d015a5199332de33faefc5619e0e90afa5e" exitCode=0 Jul 24 17:55:30.948113 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:30.948021 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" event={"ID":"11f328eb-ce9e-44ec-b12d-41850a5e7d2f","Type":"ContainerDied","Data":"c44ba5f0541dfb45d422e614ca9a7d015a5199332de33faefc5619e0e90afa5e"} Jul 24 17:55:31.035730 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.035347 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:55:31.117137 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.117107 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnk47\" (UniqueName: \"kubernetes.io/projected/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-kube-api-access-wnk47\") pod \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\" (UID: \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\") " Jul 24 17:55:31.117314 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.117153 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-openshift-service-ca\") pod \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\" (UID: \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\") " Jul 24 17:55:31.117314 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.117220 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-maas-api-tls\") pod \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\" (UID: \"11f328eb-ce9e-44ec-b12d-41850a5e7d2f\") " Jul 24 17:55:31.117594 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.117570 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "11f328eb-ce9e-44ec-b12d-41850a5e7d2f" (UID: "11f328eb-ce9e-44ec-b12d-41850a5e7d2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:55:31.121420 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.121386 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-kube-api-access-wnk47" (OuterVolumeSpecName: "kube-api-access-wnk47") pod "11f328eb-ce9e-44ec-b12d-41850a5e7d2f" (UID: "11f328eb-ce9e-44ec-b12d-41850a5e7d2f"). InnerVolumeSpecName "kube-api-access-wnk47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:55:31.121715 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.121686 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "11f328eb-ce9e-44ec-b12d-41850a5e7d2f" (UID: "11f328eb-ce9e-44ec-b12d-41850a5e7d2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:55:31.144300 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.144276 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" Jul 24 17:55:31.153328 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.153304 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" Jul 24 17:55:31.218090 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.218052 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rzdj\" (UniqueName: \"kubernetes.io/projected/ace0e865-eb2f-4cae-9205-0c30017ba0a5-kube-api-access-4rzdj\") pod \"ace0e865-eb2f-4cae-9205-0c30017ba0a5\" (UID: \"ace0e865-eb2f-4cae-9205-0c30017ba0a5\") " Jul 24 17:55:31.218090 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.218102 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9l4z\" (UniqueName: \"kubernetes.io/projected/562692de-ff57-4f1d-b9b5-30f8e0a52bd0-kube-api-access-v9l4z\") pod \"562692de-ff57-4f1d-b9b5-30f8e0a52bd0\" (UID: \"562692de-ff57-4f1d-b9b5-30f8e0a52bd0\") " Jul 24 17:55:31.218328 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.218188 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562692de-ff57-4f1d-b9b5-30f8e0a52bd0-plugins-config-volume\") pod \"562692de-ff57-4f1d-b9b5-30f8e0a52bd0\" (UID: \"562692de-ff57-4f1d-b9b5-30f8e0a52bd0\") " Jul 24 17:55:31.218328 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.218221 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace0e865-eb2f-4cae-9205-0c30017ba0a5-plugins-config-volume\") pod \"ace0e865-eb2f-4cae-9205-0c30017ba0a5\" (UID: \"ace0e865-eb2f-4cae-9205-0c30017ba0a5\") " Jul 24 17:55:31.218621 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.218591 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-openshift-service-ca\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:55:31.218621 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.218618 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-maas-api-tls\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:55:31.218810 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.218741 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnk47\" (UniqueName: \"kubernetes.io/projected/11f328eb-ce9e-44ec-b12d-41850a5e7d2f-kube-api-access-wnk47\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:55:31.218810 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.218698 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/562692de-ff57-4f1d-b9b5-30f8e0a52bd0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "562692de-ff57-4f1d-b9b5-30f8e0a52bd0" (UID: "562692de-ff57-4f1d-b9b5-30f8e0a52bd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:55:31.218810 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.218693 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace0e865-eb2f-4cae-9205-0c30017ba0a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ace0e865-eb2f-4cae-9205-0c30017ba0a5" (UID: "ace0e865-eb2f-4cae-9205-0c30017ba0a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:55:31.221136 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.221112 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562692de-ff57-4f1d-b9b5-30f8e0a52bd0-kube-api-access-v9l4z" (OuterVolumeSpecName: "kube-api-access-v9l4z") pod "562692de-ff57-4f1d-b9b5-30f8e0a52bd0" (UID: "562692de-ff57-4f1d-b9b5-30f8e0a52bd0"). InnerVolumeSpecName "kube-api-access-v9l4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:55:31.221136 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.221117 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace0e865-eb2f-4cae-9205-0c30017ba0a5-kube-api-access-4rzdj" (OuterVolumeSpecName: "kube-api-access-4rzdj") pod "ace0e865-eb2f-4cae-9205-0c30017ba0a5" (UID: "ace0e865-eb2f-4cae-9205-0c30017ba0a5"). InnerVolumeSpecName "kube-api-access-4rzdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:55:31.320075 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.319988 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562692de-ff57-4f1d-b9b5-30f8e0a52bd0-plugins-config-volume\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:55:31.320075 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.320018 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace0e865-eb2f-4cae-9205-0c30017ba0a5-plugins-config-volume\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:55:31.320075 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.320029 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rzdj\" (UniqueName: \"kubernetes.io/projected/ace0e865-eb2f-4cae-9205-0c30017ba0a5-kube-api-access-4rzdj\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:55:31.320075 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.320037 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9l4z\" (UniqueName: \"kubernetes.io/projected/562692de-ff57-4f1d-b9b5-30f8e0a52bd0-kube-api-access-v9l4z\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 17:55:31.954688 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.954652 2559 generic.go:356] "Generic (PLEG): container finished" podID="ace0e865-eb2f-4cae-9205-0c30017ba0a5" containerID="0f55799d1095c56dfbc01df6a2c9a969e6ae962053f5b5e78ea21dbbec870b1f" exitCode=0 Jul 24 17:55:31.955107 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.954697 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" Jul 24 17:55:31.955107 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.954735 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" event={"ID":"ace0e865-eb2f-4cae-9205-0c30017ba0a5","Type":"ContainerDied","Data":"0f55799d1095c56dfbc01df6a2c9a969e6ae962053f5b5e78ea21dbbec870b1f"} Jul 24 17:55:31.955107 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.954788 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487" event={"ID":"ace0e865-eb2f-4cae-9205-0c30017ba0a5","Type":"ContainerDied","Data":"448cdc742254e129d836c09845f7515790bdaf8a2e25a7ee28331f2729b69c10"} Jul 24 17:55:31.955107 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.954830 2559 scope.go:117] "RemoveContainer" containerID="0f55799d1095c56dfbc01df6a2c9a969e6ae962053f5b5e78ea21dbbec870b1f" Jul 24 17:55:31.956408 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.956348 2559 generic.go:356] "Generic (PLEG): container finished" podID="562692de-ff57-4f1d-b9b5-30f8e0a52bd0" containerID="b950e52f929f94a5754adf980fadba9adbe976ca1425d55e2f823a0f1923042a" exitCode=0 Jul 24 17:55:31.956479 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.956417 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" Jul 24 17:55:31.956554 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.956466 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" event={"ID":"562692de-ff57-4f1d-b9b5-30f8e0a52bd0","Type":"ContainerDied","Data":"b950e52f929f94a5754adf980fadba9adbe976ca1425d55e2f823a0f1923042a"} Jul 24 17:55:31.956624 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.956565 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5" event={"ID":"562692de-ff57-4f1d-b9b5-30f8e0a52bd0","Type":"ContainerDied","Data":"5026e6f4182e908e2fc9c9198ecbcc4ba1baa99920c944ed2cd7ff7f463d066e"} Jul 24 17:55:31.957896 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.957875 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" event={"ID":"11f328eb-ce9e-44ec-b12d-41850a5e7d2f","Type":"ContainerDied","Data":"74fe6a5549ec39937ea7c1f167832bbf088fed9afdbe1b17df41a7be1f96304c"} Jul 24 17:55:31.958003 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.957900 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w" Jul 24 17:55:31.966318 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.966293 2559 scope.go:117] "RemoveContainer" containerID="0f55799d1095c56dfbc01df6a2c9a969e6ae962053f5b5e78ea21dbbec870b1f" Jul 24 17:55:31.966894 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:55:31.966851 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f55799d1095c56dfbc01df6a2c9a969e6ae962053f5b5e78ea21dbbec870b1f\": container with ID starting with 0f55799d1095c56dfbc01df6a2c9a969e6ae962053f5b5e78ea21dbbec870b1f not found: ID does not exist" containerID="0f55799d1095c56dfbc01df6a2c9a969e6ae962053f5b5e78ea21dbbec870b1f" Jul 24 17:55:31.967066 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.967035 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f55799d1095c56dfbc01df6a2c9a969e6ae962053f5b5e78ea21dbbec870b1f"} err="failed to get container status \"0f55799d1095c56dfbc01df6a2c9a969e6ae962053f5b5e78ea21dbbec870b1f\": rpc error: code = NotFound desc = could not find container \"0f55799d1095c56dfbc01df6a2c9a969e6ae962053f5b5e78ea21dbbec870b1f\": container with ID starting with 0f55799d1095c56dfbc01df6a2c9a969e6ae962053f5b5e78ea21dbbec870b1f not found: ID does not exist" Jul 24 17:55:31.967154 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.967143 2559 scope.go:117] "RemoveContainer" containerID="b950e52f929f94a5754adf980fadba9adbe976ca1425d55e2f823a0f1923042a" Jul 24 17:55:31.981306 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.981283 2559 scope.go:117] "RemoveContainer" containerID="b950e52f929f94a5754adf980fadba9adbe976ca1425d55e2f823a0f1923042a" Jul 24 17:55:31.981607 ip-10-0-137-105 kubenswrapper[2559]: E0724 17:55:31.981585 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b950e52f929f94a5754adf980fadba9adbe976ca1425d55e2f823a0f1923042a\": container with ID starting with b950e52f929f94a5754adf980fadba9adbe976ca1425d55e2f823a0f1923042a not found: ID does not exist" containerID="b950e52f929f94a5754adf980fadba9adbe976ca1425d55e2f823a0f1923042a" Jul 24 17:55:31.981693 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.981621 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b950e52f929f94a5754adf980fadba9adbe976ca1425d55e2f823a0f1923042a"} err="failed to get container status \"b950e52f929f94a5754adf980fadba9adbe976ca1425d55e2f823a0f1923042a\": rpc error: code = NotFound desc = could not find container \"b950e52f929f94a5754adf980fadba9adbe976ca1425d55e2f823a0f1923042a\": container with ID starting with b950e52f929f94a5754adf980fadba9adbe976ca1425d55e2f823a0f1923042a not found: ID does not exist" Jul 24 17:55:31.981693 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.981644 2559 scope.go:117] "RemoveContainer" containerID="c44ba5f0541dfb45d422e614ca9a7d015a5199332de33faefc5619e0e90afa5e" Jul 24 17:55:31.996351 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.996325 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5"] Jul 24 17:55:31.998982 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:31.998961 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5"] Jul 24 17:55:32.015124 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:32.015073 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w"] Jul 24 17:55:32.021287 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:32.021262 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-54f231e4-845647678c-m6w5w"] Jul 24 17:55:32.054796 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:32.054710 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487"] Jul 24 17:55:32.066909 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:32.066883 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487"] Jul 24 17:55:32.942714 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:32.942681 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11f328eb-ce9e-44ec-b12d-41850a5e7d2f" path="/var/lib/kubelet/pods/11f328eb-ce9e-44ec-b12d-41850a5e7d2f/volumes" Jul 24 17:55:32.943169 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:32.943148 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="562692de-ff57-4f1d-b9b5-30f8e0a52bd0" path="/var/lib/kubelet/pods/562692de-ff57-4f1d-b9b5-30f8e0a52bd0/volumes" Jul 24 17:55:32.943539 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:32.943504 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ace0e865-eb2f-4cae-9205-0c30017ba0a5" path="/var/lib/kubelet/pods/ace0e865-eb2f-4cae-9205-0c30017ba0a5/volumes" Jul 24 17:55:38.906423 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.906389 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q"] Jul 24 17:55:38.906881 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.906719 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="562692de-ff57-4f1d-b9b5-30f8e0a52bd0" containerName="payload-pre-processing" Jul 24 17:55:38.906881 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.906732 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="562692de-ff57-4f1d-b9b5-30f8e0a52bd0" containerName="payload-pre-processing" Jul 24 17:55:38.906881 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.906750 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ace0e865-eb2f-4cae-9205-0c30017ba0a5" containerName="payload-processing" Jul 24 17:55:38.906881 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.906755 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ace0e865-eb2f-4cae-9205-0c30017ba0a5" containerName="payload-processing" Jul 24 17:55:38.906881 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.906764 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f328eb-ce9e-44ec-b12d-41850a5e7d2f" containerName="maas-api" Jul 24 17:55:38.906881 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.906771 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f328eb-ce9e-44ec-b12d-41850a5e7d2f" containerName="maas-api" Jul 24 17:55:38.906881 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.906826 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="562692de-ff57-4f1d-b9b5-30f8e0a52bd0" containerName="payload-pre-processing" Jul 24 17:55:38.906881 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.906833 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ace0e865-eb2f-4cae-9205-0c30017ba0a5" containerName="payload-processing" Jul 24 17:55:38.906881 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.906839 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f328eb-ce9e-44ec-b12d-41850a5e7d2f" containerName="maas-api" Jul 24 17:55:38.909853 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.909836 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 17:55:38.912750 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.912726 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-20962f79-serving-cert\"" Jul 24 17:55:38.935864 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.935829 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q"] Jul 24 17:55:38.986805 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.986768 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/223828c7-88ce-40d9-b8f2-03d8215d2b83-openshift-service-ca\") pod \"maas-api-e2e-ait-20962f79-fdcc878bf-wz84q\" (UID: \"223828c7-88ce-40d9-b8f2-03d8215d2b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 17:55:38.986971 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.986899 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9z29\" (UniqueName: \"kubernetes.io/projected/223828c7-88ce-40d9-b8f2-03d8215d2b83-kube-api-access-k9z29\") pod \"maas-api-e2e-ait-20962f79-fdcc878bf-wz84q\" (UID: \"223828c7-88ce-40d9-b8f2-03d8215d2b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 17:55:38.986971 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:38.986961 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/223828c7-88ce-40d9-b8f2-03d8215d2b83-maas-api-tls\") pod \"maas-api-e2e-ait-20962f79-fdcc878bf-wz84q\" (UID: \"223828c7-88ce-40d9-b8f2-03d8215d2b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 17:55:39.087772 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.087737 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/223828c7-88ce-40d9-b8f2-03d8215d2b83-maas-api-tls\") pod \"maas-api-e2e-ait-20962f79-fdcc878bf-wz84q\" (UID: \"223828c7-88ce-40d9-b8f2-03d8215d2b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 17:55:39.087772 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.087771 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/223828c7-88ce-40d9-b8f2-03d8215d2b83-openshift-service-ca\") pod \"maas-api-e2e-ait-20962f79-fdcc878bf-wz84q\" (UID: \"223828c7-88ce-40d9-b8f2-03d8215d2b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 17:55:39.087979 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.087836 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9z29\" (UniqueName: \"kubernetes.io/projected/223828c7-88ce-40d9-b8f2-03d8215d2b83-kube-api-access-k9z29\") pod \"maas-api-e2e-ait-20962f79-fdcc878bf-wz84q\" (UID: \"223828c7-88ce-40d9-b8f2-03d8215d2b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 17:55:39.088566 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.088538 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/223828c7-88ce-40d9-b8f2-03d8215d2b83-openshift-service-ca\") pod \"maas-api-e2e-ait-20962f79-fdcc878bf-wz84q\" (UID: \"223828c7-88ce-40d9-b8f2-03d8215d2b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 17:55:39.090244 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.090227 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/223828c7-88ce-40d9-b8f2-03d8215d2b83-maas-api-tls\") pod \"maas-api-e2e-ait-20962f79-fdcc878bf-wz84q\" (UID: \"223828c7-88ce-40d9-b8f2-03d8215d2b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 17:55:39.098206 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.098181 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9z29\" (UniqueName: \"kubernetes.io/projected/223828c7-88ce-40d9-b8f2-03d8215d2b83-kube-api-access-k9z29\") pod \"maas-api-e2e-ait-20962f79-fdcc878bf-wz84q\" (UID: \"223828c7-88ce-40d9-b8f2-03d8215d2b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 17:55:39.219741 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.219650 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 17:55:39.383395 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.383366 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q"] Jul 24 17:55:39.386045 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:55:39.386019 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223828c7_88ce_40d9_b8f2_03d8215d2b83.slice/crio-804ce5da4d07d49fd2899eee30f61b687645cad22268f01809b6659da4077f5f WatchSource:0}: Error finding container 804ce5da4d07d49fd2899eee30f61b687645cad22268f01809b6659da4077f5f: Status 404 returned error can't find the container with id 804ce5da4d07d49fd2899eee30f61b687645cad22268f01809b6659da4077f5f Jul 24 17:55:39.865678 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.865649 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq"] Jul 24 17:55:39.869556 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.869539 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" Jul 24 17:55:39.874347 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.874313 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-20962f79\"" Jul 24 17:55:39.874464 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.874431 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-20962f79-dockercfg-ftv27\"" Jul 24 17:55:39.885959 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.885938 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq"] Jul 24 17:55:39.944281 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.944251 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm"] Jul 24 17:55:39.947400 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.947385 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" Jul 24 17:55:39.995686 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.995655 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24948366-58d8-48ee-81d9-615501e32b09-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq\" (UID: \"24948366-58d8-48ee-81d9-615501e32b09\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" Jul 24 17:55:39.995870 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.995695 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6cqx\" (UniqueName: \"kubernetes.io/projected/7ff7a1bc-ea25-4854-810a-c3fd2e8376c9-kube-api-access-d6cqx\") pod \"payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm\" (UID: \"7ff7a1bc-ea25-4854-810a-c3fd2e8376c9\") " pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" Jul 24 17:55:39.995870 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.995728 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8njt8\" (UniqueName: \"kubernetes.io/projected/24948366-58d8-48ee-81d9-615501e32b09-kube-api-access-8njt8\") pod \"payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq\" (UID: \"24948366-58d8-48ee-81d9-615501e32b09\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" Jul 24 17:55:39.995993 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.995867 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff7a1bc-ea25-4854-810a-c3fd2e8376c9-plugins-config-volume\") pod \"payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm\" (UID: \"7ff7a1bc-ea25-4854-810a-c3fd2e8376c9\") " pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" Jul 24 17:55:39.999322 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:39.999295 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" event={"ID":"223828c7-88ce-40d9-b8f2-03d8215d2b83","Type":"ContainerStarted","Data":"804ce5da4d07d49fd2899eee30f61b687645cad22268f01809b6659da4077f5f"} Jul 24 17:55:40.024649 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:40.024624 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm"] Jul 24 17:55:40.097217 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:40.097170 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24948366-58d8-48ee-81d9-615501e32b09-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq\" (UID: \"24948366-58d8-48ee-81d9-615501e32b09\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" Jul 24 17:55:40.097217 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:40.097212 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6cqx\" (UniqueName: \"kubernetes.io/projected/7ff7a1bc-ea25-4854-810a-c3fd2e8376c9-kube-api-access-d6cqx\") pod \"payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm\" (UID: \"7ff7a1bc-ea25-4854-810a-c3fd2e8376c9\") " pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" Jul 24 17:55:40.097814 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:40.097257 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8njt8\" (UniqueName: \"kubernetes.io/projected/24948366-58d8-48ee-81d9-615501e32b09-kube-api-access-8njt8\") pod \"payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq\" (UID: \"24948366-58d8-48ee-81d9-615501e32b09\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" Jul 24 17:55:40.097814 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:40.097349 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff7a1bc-ea25-4854-810a-c3fd2e8376c9-plugins-config-volume\") pod \"payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm\" (UID: \"7ff7a1bc-ea25-4854-810a-c3fd2e8376c9\") " pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" Jul 24 17:55:40.098085 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:40.098061 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24948366-58d8-48ee-81d9-615501e32b09-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq\" (UID: \"24948366-58d8-48ee-81d9-615501e32b09\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" Jul 24 17:55:40.098153 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:40.098138 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff7a1bc-ea25-4854-810a-c3fd2e8376c9-plugins-config-volume\") pod \"payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm\" (UID: \"7ff7a1bc-ea25-4854-810a-c3fd2e8376c9\") " pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" Jul 24 17:55:40.113242 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:40.113198 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8njt8\" (UniqueName: \"kubernetes.io/projected/24948366-58d8-48ee-81d9-615501e32b09-kube-api-access-8njt8\") pod \"payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq\" (UID: \"24948366-58d8-48ee-81d9-615501e32b09\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" Jul 24 17:55:40.127206 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:40.127132 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6cqx\" (UniqueName: \"kubernetes.io/projected/7ff7a1bc-ea25-4854-810a-c3fd2e8376c9-kube-api-access-d6cqx\") pod \"payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm\" (UID: \"7ff7a1bc-ea25-4854-810a-c3fd2e8376c9\") " pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" Jul 24 17:55:40.180111 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:40.180068 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" Jul 24 17:55:40.257302 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:40.257270 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" Jul 24 17:55:40.349807 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:40.349754 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq"] Jul 24 17:55:40.352745 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:55:40.352709 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24948366_58d8_48ee_81d9_615501e32b09.slice/crio-ad9d7e08ad68209a8f60d65308a51f1c21f88aebacc709610c9745780141d1d1 WatchSource:0}: Error finding container ad9d7e08ad68209a8f60d65308a51f1c21f88aebacc709610c9745780141d1d1: Status 404 returned error can't find the container with id ad9d7e08ad68209a8f60d65308a51f1c21f88aebacc709610c9745780141d1d1 Jul 24 17:55:40.459538 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:40.459491 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm"] Jul 24 17:55:40.461941 ip-10-0-137-105 kubenswrapper[2559]: W0724 17:55:40.461916 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff7a1bc_ea25_4854_810a_c3fd2e8376c9.slice/crio-8b02a7feaa189e3641c807a9dc2d594509aacce0cfa3a00165863de72ab2cc21 WatchSource:0}: Error finding container 8b02a7feaa189e3641c807a9dc2d594509aacce0cfa3a00165863de72ab2cc21: Status 404 returned error can't find the container with id 8b02a7feaa189e3641c807a9dc2d594509aacce0cfa3a00165863de72ab2cc21 Jul 24 17:55:41.012090 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:41.012051 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" event={"ID":"7ff7a1bc-ea25-4854-810a-c3fd2e8376c9","Type":"ContainerStarted","Data":"e3b166e9328ebb58ceb643865f5fb0b2b2301cf1e723a2954c415d037b5e9767"} Jul 24 17:55:41.012621 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:41.012154 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" event={"ID":"7ff7a1bc-ea25-4854-810a-c3fd2e8376c9","Type":"ContainerStarted","Data":"8b02a7feaa189e3641c807a9dc2d594509aacce0cfa3a00165863de72ab2cc21"} Jul 24 17:55:41.012621 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:41.012324 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" Jul 24 17:55:41.013979 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:41.013953 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" event={"ID":"223828c7-88ce-40d9-b8f2-03d8215d2b83","Type":"ContainerStarted","Data":"1e970b777df34568cb111ec4f8b337beb654b8b32efe703e67887a4e03fc2875"} Jul 24 17:55:41.014289 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:41.014272 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 17:55:41.015577 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:41.015558 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" event={"ID":"24948366-58d8-48ee-81d9-615501e32b09","Type":"ContainerStarted","Data":"e3c4aff6f39da4cb11ae642550ce455b3de2d1a291e12f24bc737af3aa407574"} Jul 24 17:55:41.015727 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:41.015598 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" event={"ID":"24948366-58d8-48ee-81d9-615501e32b09","Type":"ContainerStarted","Data":"ad9d7e08ad68209a8f60d65308a51f1c21f88aebacc709610c9745780141d1d1"} Jul 24 17:55:41.015727 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:41.015641 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" Jul 24 17:55:41.088120 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:41.088053 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" podStartSLOduration=2.0880357 podStartE2EDuration="2.0880357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:55:41.087664243 +0000 UTC m=+2138.739555716" watchObservedRunningTime="2026-07-24 17:55:41.0880357 +0000 UTC m=+2138.739927173" Jul 24 17:55:41.088299 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:41.088173 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" podStartSLOduration=2.08816473 podStartE2EDuration="2.08816473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:55:41.047231837 +0000 UTC m=+2138.699123309" watchObservedRunningTime="2026-07-24 17:55:41.08816473 +0000 UTC m=+2138.740056204" Jul 24 17:55:41.137394 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:41.137346 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" podStartSLOduration=2.540464455 podStartE2EDuration="3.137328156s" totalImagesPullingTime="596.863701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:55:41.136155308 +0000 UTC m=+2138.788046781" watchObservedRunningTime="2026-07-24 17:55:41.137328156 +0000 UTC m=+2138.789219633" Jul 24 17:55:48.030838 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:48.030812 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 17:55:52.022971 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:52.022945 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" Jul 24 17:55:52.023357 ip-10-0-137-105 kubenswrapper[2559]: I0724 17:55:52.023182 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" Jul 24 18:00:55.960876 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:55.960841 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q"] Jul 24 18:00:55.961446 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:55.961084 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" podUID="223828c7-88ce-40d9-b8f2-03d8215d2b83" containerName="maas-api" containerID="cri-o://1e970b777df34568cb111ec4f8b337beb654b8b32efe703e67887a4e03fc2875" gracePeriod=30 Jul 24 18:00:56.029604 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.029569 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm"] Jul 24 18:00:56.029886 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.029839 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" podUID="7ff7a1bc-ea25-4854-810a-c3fd2e8376c9" containerName="payload-processing" containerID="cri-o://e3b166e9328ebb58ceb643865f5fb0b2b2301cf1e723a2954c415d037b5e9767" gracePeriod=30 Jul 24 18:00:56.038163 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.038135 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq"] Jul 24 18:00:56.038462 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.038434 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" podUID="24948366-58d8-48ee-81d9-615501e32b09" containerName="payload-pre-processing" containerID="cri-o://e3c4aff6f39da4cb11ae642550ce455b3de2d1a291e12f24bc737af3aa407574" gracePeriod=30 Jul 24 18:00:56.311935 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.311910 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" Jul 24 18:00:56.314865 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.314843 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" Jul 24 18:00:56.320643 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.320623 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 18:00:56.384053 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.382687 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/223828c7-88ce-40d9-b8f2-03d8215d2b83-maas-api-tls\") pod \"223828c7-88ce-40d9-b8f2-03d8215d2b83\" (UID: \"223828c7-88ce-40d9-b8f2-03d8215d2b83\") " Jul 24 18:00:56.384053 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.382747 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24948366-58d8-48ee-81d9-615501e32b09-plugins-config-volume\") pod \"24948366-58d8-48ee-81d9-615501e32b09\" (UID: \"24948366-58d8-48ee-81d9-615501e32b09\") " Jul 24 18:00:56.384053 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.382782 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9z29\" (UniqueName: \"kubernetes.io/projected/223828c7-88ce-40d9-b8f2-03d8215d2b83-kube-api-access-k9z29\") pod \"223828c7-88ce-40d9-b8f2-03d8215d2b83\" (UID: \"223828c7-88ce-40d9-b8f2-03d8215d2b83\") " Jul 24 18:00:56.384053 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.382860 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/223828c7-88ce-40d9-b8f2-03d8215d2b83-openshift-service-ca\") pod \"223828c7-88ce-40d9-b8f2-03d8215d2b83\" (UID: \"223828c7-88ce-40d9-b8f2-03d8215d2b83\") " Jul 24 18:00:56.384053 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.382894 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6cqx\" (UniqueName: \"kubernetes.io/projected/7ff7a1bc-ea25-4854-810a-c3fd2e8376c9-kube-api-access-d6cqx\") pod \"7ff7a1bc-ea25-4854-810a-c3fd2e8376c9\" (UID: \"7ff7a1bc-ea25-4854-810a-c3fd2e8376c9\") " Jul 24 18:00:56.384053 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.382919 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8njt8\" (UniqueName: \"kubernetes.io/projected/24948366-58d8-48ee-81d9-615501e32b09-kube-api-access-8njt8\") pod \"24948366-58d8-48ee-81d9-615501e32b09\" (UID: \"24948366-58d8-48ee-81d9-615501e32b09\") " Jul 24 18:00:56.384053 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.382958 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff7a1bc-ea25-4854-810a-c3fd2e8376c9-plugins-config-volume\") pod \"7ff7a1bc-ea25-4854-810a-c3fd2e8376c9\" (UID: \"7ff7a1bc-ea25-4854-810a-c3fd2e8376c9\") " Jul 24 18:00:56.384053 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.383748 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff7a1bc-ea25-4854-810a-c3fd2e8376c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ff7a1bc-ea25-4854-810a-c3fd2e8376c9" (UID: "7ff7a1bc-ea25-4854-810a-c3fd2e8376c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:00:56.387592 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.387556 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223828c7-88ce-40d9-b8f2-03d8215d2b83-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "223828c7-88ce-40d9-b8f2-03d8215d2b83" (UID: "223828c7-88ce-40d9-b8f2-03d8215d2b83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:00:56.388005 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.387976 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24948366-58d8-48ee-81d9-615501e32b09-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24948366-58d8-48ee-81d9-615501e32b09" (UID: "24948366-58d8-48ee-81d9-615501e32b09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:00:56.389960 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.389927 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24948366-58d8-48ee-81d9-615501e32b09-kube-api-access-8njt8" (OuterVolumeSpecName: "kube-api-access-8njt8") pod "24948366-58d8-48ee-81d9-615501e32b09" (UID: "24948366-58d8-48ee-81d9-615501e32b09"). InnerVolumeSpecName "kube-api-access-8njt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:00:56.390081 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.389962 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff7a1bc-ea25-4854-810a-c3fd2e8376c9-kube-api-access-d6cqx" (OuterVolumeSpecName: "kube-api-access-d6cqx") pod "7ff7a1bc-ea25-4854-810a-c3fd2e8376c9" (UID: "7ff7a1bc-ea25-4854-810a-c3fd2e8376c9"). InnerVolumeSpecName "kube-api-access-d6cqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:00:56.390081 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.390037 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223828c7-88ce-40d9-b8f2-03d8215d2b83-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "223828c7-88ce-40d9-b8f2-03d8215d2b83" (UID: "223828c7-88ce-40d9-b8f2-03d8215d2b83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:00:56.390422 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.390406 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223828c7-88ce-40d9-b8f2-03d8215d2b83-kube-api-access-k9z29" (OuterVolumeSpecName: "kube-api-access-k9z29") pod "223828c7-88ce-40d9-b8f2-03d8215d2b83" (UID: "223828c7-88ce-40d9-b8f2-03d8215d2b83"). InnerVolumeSpecName "kube-api-access-k9z29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:00:56.401698 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.401668 2559 generic.go:356] "Generic (PLEG): container finished" podID="24948366-58d8-48ee-81d9-615501e32b09" containerID="e3c4aff6f39da4cb11ae642550ce455b3de2d1a291e12f24bc737af3aa407574" exitCode=0 Jul 24 18:00:56.401822 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.401714 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" event={"ID":"24948366-58d8-48ee-81d9-615501e32b09","Type":"ContainerDied","Data":"e3c4aff6f39da4cb11ae642550ce455b3de2d1a291e12f24bc737af3aa407574"} Jul 24 18:00:56.401822 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.401767 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" Jul 24 18:00:56.401822 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.401812 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq" event={"ID":"24948366-58d8-48ee-81d9-615501e32b09","Type":"ContainerDied","Data":"ad9d7e08ad68209a8f60d65308a51f1c21f88aebacc709610c9745780141d1d1"} Jul 24 18:00:56.401981 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.401831 2559 scope.go:117] "RemoveContainer" containerID="e3c4aff6f39da4cb11ae642550ce455b3de2d1a291e12f24bc737af3aa407574" Jul 24 18:00:56.403192 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.403115 2559 generic.go:356] "Generic (PLEG): container finished" podID="7ff7a1bc-ea25-4854-810a-c3fd2e8376c9" containerID="e3b166e9328ebb58ceb643865f5fb0b2b2301cf1e723a2954c415d037b5e9767" exitCode=0 Jul 24 18:00:56.403303 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.403228 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" Jul 24 18:00:56.403368 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.403227 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" event={"ID":"7ff7a1bc-ea25-4854-810a-c3fd2e8376c9","Type":"ContainerDied","Data":"e3b166e9328ebb58ceb643865f5fb0b2b2301cf1e723a2954c415d037b5e9767"} Jul 24 18:00:56.403368 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.403358 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm" event={"ID":"7ff7a1bc-ea25-4854-810a-c3fd2e8376c9","Type":"ContainerDied","Data":"8b02a7feaa189e3641c807a9dc2d594509aacce0cfa3a00165863de72ab2cc21"} Jul 24 18:00:56.404835 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.404811 2559 generic.go:356] "Generic (PLEG): container finished" podID="223828c7-88ce-40d9-b8f2-03d8215d2b83" containerID="1e970b777df34568cb111ec4f8b337beb654b8b32efe703e67887a4e03fc2875" exitCode=0 Jul 24 18:00:56.404937 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.404870 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" event={"ID":"223828c7-88ce-40d9-b8f2-03d8215d2b83","Type":"ContainerDied","Data":"1e970b777df34568cb111ec4f8b337beb654b8b32efe703e67887a4e03fc2875"} Jul 24 18:00:56.404937 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.404885 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" Jul 24 18:00:56.404937 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.404909 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q" event={"ID":"223828c7-88ce-40d9-b8f2-03d8215d2b83","Type":"ContainerDied","Data":"804ce5da4d07d49fd2899eee30f61b687645cad22268f01809b6659da4077f5f"} Jul 24 18:00:56.412709 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.412691 2559 scope.go:117] "RemoveContainer" containerID="e3c4aff6f39da4cb11ae642550ce455b3de2d1a291e12f24bc737af3aa407574" Jul 24 18:00:56.413214 ip-10-0-137-105 kubenswrapper[2559]: E0724 18:00:56.413186 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c4aff6f39da4cb11ae642550ce455b3de2d1a291e12f24bc737af3aa407574\": container with ID starting with e3c4aff6f39da4cb11ae642550ce455b3de2d1a291e12f24bc737af3aa407574 not found: ID does not exist" containerID="e3c4aff6f39da4cb11ae642550ce455b3de2d1a291e12f24bc737af3aa407574" Jul 24 18:00:56.413321 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.413223 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c4aff6f39da4cb11ae642550ce455b3de2d1a291e12f24bc737af3aa407574"} err="failed to get container status \"e3c4aff6f39da4cb11ae642550ce455b3de2d1a291e12f24bc737af3aa407574\": rpc error: code = NotFound desc = could not find container \"e3c4aff6f39da4cb11ae642550ce455b3de2d1a291e12f24bc737af3aa407574\": container with ID starting with e3c4aff6f39da4cb11ae642550ce455b3de2d1a291e12f24bc737af3aa407574 not found: ID does not exist" Jul 24 18:00:56.413321 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.413246 2559 scope.go:117] "RemoveContainer" containerID="e3b166e9328ebb58ceb643865f5fb0b2b2301cf1e723a2954c415d037b5e9767" Jul 24 18:00:56.421350 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.421335 2559 scope.go:117] "RemoveContainer" containerID="e3b166e9328ebb58ceb643865f5fb0b2b2301cf1e723a2954c415d037b5e9767" Jul 24 18:00:56.421649 ip-10-0-137-105 kubenswrapper[2559]: E0724 18:00:56.421631 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b166e9328ebb58ceb643865f5fb0b2b2301cf1e723a2954c415d037b5e9767\": container with ID starting with e3b166e9328ebb58ceb643865f5fb0b2b2301cf1e723a2954c415d037b5e9767 not found: ID does not exist" containerID="e3b166e9328ebb58ceb643865f5fb0b2b2301cf1e723a2954c415d037b5e9767" Jul 24 18:00:56.421700 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.421657 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b166e9328ebb58ceb643865f5fb0b2b2301cf1e723a2954c415d037b5e9767"} err="failed to get container status \"e3b166e9328ebb58ceb643865f5fb0b2b2301cf1e723a2954c415d037b5e9767\": rpc error: code = NotFound desc = could not find container \"e3b166e9328ebb58ceb643865f5fb0b2b2301cf1e723a2954c415d037b5e9767\": container with ID starting with e3b166e9328ebb58ceb643865f5fb0b2b2301cf1e723a2954c415d037b5e9767 not found: ID does not exist" Jul 24 18:00:56.421700 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.421674 2559 scope.go:117] "RemoveContainer" containerID="1e970b777df34568cb111ec4f8b337beb654b8b32efe703e67887a4e03fc2875" Jul 24 18:00:56.430830 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.430810 2559 scope.go:117] "RemoveContainer" containerID="1e970b777df34568cb111ec4f8b337beb654b8b32efe703e67887a4e03fc2875" Jul 24 18:00:56.431282 ip-10-0-137-105 kubenswrapper[2559]: E0724 18:00:56.431252 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e970b777df34568cb111ec4f8b337beb654b8b32efe703e67887a4e03fc2875\": container with ID starting with 1e970b777df34568cb111ec4f8b337beb654b8b32efe703e67887a4e03fc2875 not found: ID does not exist" containerID="1e970b777df34568cb111ec4f8b337beb654b8b32efe703e67887a4e03fc2875" Jul 24 18:00:56.431371 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.431292 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e970b777df34568cb111ec4f8b337beb654b8b32efe703e67887a4e03fc2875"} err="failed to get container status \"1e970b777df34568cb111ec4f8b337beb654b8b32efe703e67887a4e03fc2875\": rpc error: code = NotFound desc = could not find container \"1e970b777df34568cb111ec4f8b337beb654b8b32efe703e67887a4e03fc2875\": container with ID starting with 1e970b777df34568cb111ec4f8b337beb654b8b32efe703e67887a4e03fc2875 not found: ID does not exist" Jul 24 18:00:56.433000 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.432979 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm"] Jul 24 18:00:56.437873 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.437854 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm"] Jul 24 18:00:56.450250 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.450227 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq"] Jul 24 18:00:56.456443 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.456411 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq"] Jul 24 18:00:56.467807 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.467785 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q"] Jul 24 18:00:56.474611 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.474587 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20962f79-fdcc878bf-wz84q"] Jul 24 18:00:56.484462 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.484439 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9z29\" (UniqueName: \"kubernetes.io/projected/223828c7-88ce-40d9-b8f2-03d8215d2b83-kube-api-access-k9z29\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 18:00:56.484462 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.484459 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/223828c7-88ce-40d9-b8f2-03d8215d2b83-openshift-service-ca\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 18:00:56.484653 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.484471 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6cqx\" (UniqueName: \"kubernetes.io/projected/7ff7a1bc-ea25-4854-810a-c3fd2e8376c9-kube-api-access-d6cqx\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 18:00:56.484653 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.484479 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8njt8\" (UniqueName: \"kubernetes.io/projected/24948366-58d8-48ee-81d9-615501e32b09-kube-api-access-8njt8\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 18:00:56.484653 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.484488 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff7a1bc-ea25-4854-810a-c3fd2e8376c9-plugins-config-volume\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 18:00:56.484653 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.484496 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/223828c7-88ce-40d9-b8f2-03d8215d2b83-maas-api-tls\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 18:00:56.484653 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.484504 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24948366-58d8-48ee-81d9-615501e32b09-plugins-config-volume\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 18:00:56.942295 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.942253 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="223828c7-88ce-40d9-b8f2-03d8215d2b83" path="/var/lib/kubelet/pods/223828c7-88ce-40d9-b8f2-03d8215d2b83/volumes" Jul 24 18:00:56.942737 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.942717 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24948366-58d8-48ee-81d9-615501e32b09" path="/var/lib/kubelet/pods/24948366-58d8-48ee-81d9-615501e32b09/volumes" Jul 24 18:00:56.943116 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:00:56.943099 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ff7a1bc-ea25-4854-810a-c3fd2e8376c9" path="/var/lib/kubelet/pods/7ff7a1bc-ea25-4854-810a-c3fd2e8376c9/volumes" Jul 24 18:01:03.778342 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.778308 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn"] Jul 24 18:01:03.778732 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.778647 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24948366-58d8-48ee-81d9-615501e32b09" containerName="payload-pre-processing" Jul 24 18:01:03.778732 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.778659 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="24948366-58d8-48ee-81d9-615501e32b09" containerName="payload-pre-processing" Jul 24 18:01:03.778732 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.778672 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff7a1bc-ea25-4854-810a-c3fd2e8376c9" containerName="payload-processing" Jul 24 18:01:03.778732 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.778678 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff7a1bc-ea25-4854-810a-c3fd2e8376c9" containerName="payload-processing" Jul 24 18:01:03.778732 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.778690 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="223828c7-88ce-40d9-b8f2-03d8215d2b83" containerName="maas-api" Jul 24 18:01:03.778732 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.778695 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="223828c7-88ce-40d9-b8f2-03d8215d2b83" containerName="maas-api" Jul 24 18:01:03.778927 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.778742 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="223828c7-88ce-40d9-b8f2-03d8215d2b83" containerName="maas-api" Jul 24 18:01:03.778927 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.778752 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="24948366-58d8-48ee-81d9-615501e32b09" containerName="payload-pre-processing" Jul 24 18:01:03.778927 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.778759 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ff7a1bc-ea25-4854-810a-c3fd2e8376c9" containerName="payload-processing" Jul 24 18:01:03.783558 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.783538 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:01:03.786372 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.786355 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-842768d6-serving-cert\"" Jul 24 18:01:03.797492 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.797469 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn"] Jul 24 18:01:03.848444 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.848411 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20d0ecca-3730-4028-bc13-0e2e63677856-maas-api-tls\") pod \"maas-api-e2e-derive-842768d6-67b44d5446-zlpbn\" (UID: \"20d0ecca-3730-4028-bc13-0e2e63677856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:01:03.848655 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.848476 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d0ecca-3730-4028-bc13-0e2e63677856-openshift-service-ca\") pod \"maas-api-e2e-derive-842768d6-67b44d5446-zlpbn\" (UID: \"20d0ecca-3730-4028-bc13-0e2e63677856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:01:03.848732 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.848646 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw8wk\" (UniqueName: \"kubernetes.io/projected/20d0ecca-3730-4028-bc13-0e2e63677856-kube-api-access-dw8wk\") pod \"maas-api-e2e-derive-842768d6-67b44d5446-zlpbn\" (UID: \"20d0ecca-3730-4028-bc13-0e2e63677856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:01:03.949650 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.949619 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d0ecca-3730-4028-bc13-0e2e63677856-openshift-service-ca\") pod \"maas-api-e2e-derive-842768d6-67b44d5446-zlpbn\" (UID: \"20d0ecca-3730-4028-bc13-0e2e63677856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:01:03.949843 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.949662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw8wk\" (UniqueName: \"kubernetes.io/projected/20d0ecca-3730-4028-bc13-0e2e63677856-kube-api-access-dw8wk\") pod \"maas-api-e2e-derive-842768d6-67b44d5446-zlpbn\" (UID: \"20d0ecca-3730-4028-bc13-0e2e63677856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:01:03.949843 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.949694 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20d0ecca-3730-4028-bc13-0e2e63677856-maas-api-tls\") pod \"maas-api-e2e-derive-842768d6-67b44d5446-zlpbn\" (UID: \"20d0ecca-3730-4028-bc13-0e2e63677856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:01:03.950308 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.950285 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d0ecca-3730-4028-bc13-0e2e63677856-openshift-service-ca\") pod \"maas-api-e2e-derive-842768d6-67b44d5446-zlpbn\" (UID: \"20d0ecca-3730-4028-bc13-0e2e63677856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:01:03.952236 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.952216 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20d0ecca-3730-4028-bc13-0e2e63677856-maas-api-tls\") pod \"maas-api-e2e-derive-842768d6-67b44d5446-zlpbn\" (UID: \"20d0ecca-3730-4028-bc13-0e2e63677856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:01:03.959457 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:03.959431 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw8wk\" (UniqueName: \"kubernetes.io/projected/20d0ecca-3730-4028-bc13-0e2e63677856-kube-api-access-dw8wk\") pod \"maas-api-e2e-derive-842768d6-67b44d5446-zlpbn\" (UID: \"20d0ecca-3730-4028-bc13-0e2e63677856\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:01:04.093422 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.093345 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:01:04.228431 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.228407 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn"] Jul 24 18:01:04.231009 ip-10-0-137-105 kubenswrapper[2559]: W0724 18:01:04.230980 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d0ecca_3730_4028_bc13_0e2e63677856.slice/crio-bf38f80585fca49cf7586228d04eb18368b515708a8e804ede5fa55b67a6b26e WatchSource:0}: Error finding container bf38f80585fca49cf7586228d04eb18368b515708a8e804ede5fa55b67a6b26e: Status 404 returned error can't find the container with id bf38f80585fca49cf7586228d04eb18368b515708a8e804ede5fa55b67a6b26e Jul 24 18:01:04.232258 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.232240 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 18:01:04.442123 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.442043 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" event={"ID":"20d0ecca-3730-4028-bc13-0e2e63677856","Type":"ContainerStarted","Data":"bf38f80585fca49cf7586228d04eb18368b515708a8e804ede5fa55b67a6b26e"} Jul 24 18:01:04.635366 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.635324 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc"] Jul 24 18:01:04.639715 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.639693 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" Jul 24 18:01:04.642329 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.642310 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-842768d6-dockercfg-kw78k\"" Jul 24 18:01:04.642425 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.642384 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-842768d6\"" Jul 24 18:01:04.658353 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.658326 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc"] Jul 24 18:01:04.709271 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.709188 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54"] Jul 24 18:01:04.712633 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.712610 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" Jul 24 18:01:04.723984 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.723960 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54"] Jul 24 18:01:04.756590 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.756559 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f-plugins-config-volume\") pod \"payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc\" (UID: \"be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f\") " pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" Jul 24 18:01:04.756713 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.756592 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vggdq\" (UniqueName: \"kubernetes.io/projected/be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f-kube-api-access-vggdq\") pod \"payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc\" (UID: \"be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f\") " pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" Jul 24 18:01:04.857799 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.857765 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7028d9-1dd2-47b3-b99f-890f3a1e7755-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54\" (UID: \"3e7028d9-1dd2-47b3-b99f-890f3a1e7755\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" Jul 24 18:01:04.858184 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.857831 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f-plugins-config-volume\") pod \"payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc\" (UID: \"be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f\") " pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" Jul 24 18:01:04.858184 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.857862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n42h\" (UniqueName: \"kubernetes.io/projected/3e7028d9-1dd2-47b3-b99f-890f3a1e7755-kube-api-access-2n42h\") pod \"payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54\" (UID: \"3e7028d9-1dd2-47b3-b99f-890f3a1e7755\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" Jul 24 18:01:04.858184 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.857894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vggdq\" (UniqueName: \"kubernetes.io/projected/be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f-kube-api-access-vggdq\") pod \"payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc\" (UID: \"be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f\") " pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" Jul 24 18:01:04.858656 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.858638 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f-plugins-config-volume\") pod \"payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc\" (UID: \"be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f\") " pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" Jul 24 18:01:04.867469 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.867439 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vggdq\" (UniqueName: \"kubernetes.io/projected/be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f-kube-api-access-vggdq\") pod \"payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc\" (UID: \"be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f\") " pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" Jul 24 18:01:04.950227 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.950191 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" Jul 24 18:01:04.958424 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.958390 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7028d9-1dd2-47b3-b99f-890f3a1e7755-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54\" (UID: \"3e7028d9-1dd2-47b3-b99f-890f3a1e7755\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" Jul 24 18:01:04.958592 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.958448 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n42h\" (UniqueName: \"kubernetes.io/projected/3e7028d9-1dd2-47b3-b99f-890f3a1e7755-kube-api-access-2n42h\") pod \"payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54\" (UID: \"3e7028d9-1dd2-47b3-b99f-890f3a1e7755\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" Jul 24 18:01:04.959240 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.959219 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7028d9-1dd2-47b3-b99f-890f3a1e7755-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54\" (UID: \"3e7028d9-1dd2-47b3-b99f-890f3a1e7755\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" Jul 24 18:01:04.967981 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:04.967960 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n42h\" (UniqueName: \"kubernetes.io/projected/3e7028d9-1dd2-47b3-b99f-890f3a1e7755-kube-api-access-2n42h\") pod \"payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54\" (UID: \"3e7028d9-1dd2-47b3-b99f-890f3a1e7755\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" Jul 24 18:01:05.023259 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:05.023234 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" Jul 24 18:01:05.095439 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:05.095140 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc"] Jul 24 18:01:05.098166 ip-10-0-137-105 kubenswrapper[2559]: W0724 18:01:05.098094 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2ee6c7_7a33_4d5e_b9bb_1df9cec7be6f.slice/crio-b4368d484cd1e7263b358684c24dcc18b0d5e63b786b58b41b623edb494ee9c5 WatchSource:0}: Error finding container b4368d484cd1e7263b358684c24dcc18b0d5e63b786b58b41b623edb494ee9c5: Status 404 returned error can't find the container with id b4368d484cd1e7263b358684c24dcc18b0d5e63b786b58b41b623edb494ee9c5 Jul 24 18:01:05.185013 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:05.184987 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54"] Jul 24 18:01:05.187762 ip-10-0-137-105 kubenswrapper[2559]: W0724 18:01:05.187734 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7028d9_1dd2_47b3_b99f_890f3a1e7755.slice/crio-5af96fbd0d0f2984df8bf29462db4906df5bb46b6683938f97fc265be1fe1c8b WatchSource:0}: Error finding container 5af96fbd0d0f2984df8bf29462db4906df5bb46b6683938f97fc265be1fe1c8b: Status 404 returned error can't find the container with id 5af96fbd0d0f2984df8bf29462db4906df5bb46b6683938f97fc265be1fe1c8b Jul 24 18:01:05.447608 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:05.447567 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" event={"ID":"3e7028d9-1dd2-47b3-b99f-890f3a1e7755","Type":"ContainerStarted","Data":"3a5fd46ed8564f5fd44b4849f72ac45971c35bb38bdb38e7c570ef7c28a9962c"} Jul 24 18:01:05.447798 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:05.447669 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" event={"ID":"3e7028d9-1dd2-47b3-b99f-890f3a1e7755","Type":"ContainerStarted","Data":"5af96fbd0d0f2984df8bf29462db4906df5bb46b6683938f97fc265be1fe1c8b"} Jul 24 18:01:05.447798 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:05.447734 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" Jul 24 18:01:05.449218 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:05.449190 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" event={"ID":"20d0ecca-3730-4028-bc13-0e2e63677856","Type":"ContainerStarted","Data":"348c7336f6be5bd7c59e0fa8009c7a5b61055a6c86d801bb19cbd31e60a8b76f"} Jul 24 18:01:05.449347 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:05.449245 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:01:05.450660 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:05.450638 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" event={"ID":"be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f","Type":"ContainerStarted","Data":"58efa987cb206f75e2389df4d5d6e1d37d19d35517239f9136c01282e2212018"} Jul 24 18:01:05.450773 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:05.450680 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" event={"ID":"be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f","Type":"ContainerStarted","Data":"b4368d484cd1e7263b358684c24dcc18b0d5e63b786b58b41b623edb494ee9c5"} Jul 24 18:01:05.450773 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:05.450727 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" Jul 24 18:01:05.472396 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:05.472314 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" podStartSLOduration=1.472301259 podStartE2EDuration="1.472301259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:01:05.471011901 +0000 UTC m=+2463.122903371" watchObservedRunningTime="2026-07-24 18:01:05.472301259 +0000 UTC m=+2463.124192773" Jul 24 18:01:05.505920 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:05.505879 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" podStartSLOduration=1.505861898 podStartE2EDuration="1.505861898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:01:05.504310808 +0000 UTC m=+2463.156202283" watchObservedRunningTime="2026-07-24 18:01:05.505861898 +0000 UTC m=+2463.157753371" Jul 24 18:01:05.539126 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:05.539075 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" podStartSLOduration=1.755464177 podStartE2EDuration="2.539056411s" totalImagesPullingTime="783.592234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:01:05.538359657 +0000 UTC m=+2463.190251135" watchObservedRunningTime="2026-07-24 18:01:05.539056411 +0000 UTC m=+2463.190947886" Jul 24 18:01:11.459684 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:11.459654 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:01:16.456424 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:16.456366 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" podUID="3e7028d9-1dd2-47b3-b99f-890f3a1e7755" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.48:9004: connect: connection refused" Jul 24 18:01:16.457018 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:16.457000 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" Jul 24 18:01:26.455420 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:01:26.455390 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" Jul 24 18:06:20.713090 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:20.713056 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn"] Jul 24 18:06:20.713629 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:20.713373 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" podUID="20d0ecca-3730-4028-bc13-0e2e63677856" containerName="maas-api" containerID="cri-o://348c7336f6be5bd7c59e0fa8009c7a5b61055a6c86d801bb19cbd31e60a8b76f" gracePeriod=30 Jul 24 18:06:20.792603 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:20.792570 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc"] Jul 24 18:06:20.792820 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:20.792797 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" podUID="be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f" containerName="payload-processing" containerID="cri-o://58efa987cb206f75e2389df4d5d6e1d37d19d35517239f9136c01282e2212018" gracePeriod=30 Jul 24 18:06:20.805265 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:20.805224 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54"] Jul 24 18:06:20.805535 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:20.805461 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" podUID="3e7028d9-1dd2-47b3-b99f-890f3a1e7755" containerName="payload-pre-processing" containerID="cri-o://3a5fd46ed8564f5fd44b4849f72ac45971c35bb38bdb38e7c570ef7c28a9962c" gracePeriod=30 Jul 24 18:06:20.834472 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:20.834443 2559 generic.go:356] "Generic (PLEG): container finished" podID="20d0ecca-3730-4028-bc13-0e2e63677856" containerID="348c7336f6be5bd7c59e0fa8009c7a5b61055a6c86d801bb19cbd31e60a8b76f" exitCode=0 Jul 24 18:06:20.834639 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:20.834543 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" event={"ID":"20d0ecca-3730-4028-bc13-0e2e63677856","Type":"ContainerDied","Data":"348c7336f6be5bd7c59e0fa8009c7a5b61055a6c86d801bb19cbd31e60a8b76f"} Jul 24 18:06:21.056670 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.056644 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:06:21.077602 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.077575 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" Jul 24 18:06:21.089137 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.089114 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" Jul 24 18:06:21.182984 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.182948 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vggdq\" (UniqueName: \"kubernetes.io/projected/be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f-kube-api-access-vggdq\") pod \"be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f\" (UID: \"be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f\") " Jul 24 18:06:21.182984 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.182982 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n42h\" (UniqueName: \"kubernetes.io/projected/3e7028d9-1dd2-47b3-b99f-890f3a1e7755-kube-api-access-2n42h\") pod \"3e7028d9-1dd2-47b3-b99f-890f3a1e7755\" (UID: \"3e7028d9-1dd2-47b3-b99f-890f3a1e7755\") " Jul 24 18:06:21.183214 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.183127 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f-plugins-config-volume\") pod \"be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f\" (UID: \"be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f\") " Jul 24 18:06:21.183272 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.183217 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw8wk\" (UniqueName: \"kubernetes.io/projected/20d0ecca-3730-4028-bc13-0e2e63677856-kube-api-access-dw8wk\") pod \"20d0ecca-3730-4028-bc13-0e2e63677856\" (UID: \"20d0ecca-3730-4028-bc13-0e2e63677856\") " Jul 24 18:06:21.183272 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.183259 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7028d9-1dd2-47b3-b99f-890f3a1e7755-plugins-config-volume\") pod \"3e7028d9-1dd2-47b3-b99f-890f3a1e7755\" (UID: \"3e7028d9-1dd2-47b3-b99f-890f3a1e7755\") " Jul 24 18:06:21.183371 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.183332 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20d0ecca-3730-4028-bc13-0e2e63677856-maas-api-tls\") pod \"20d0ecca-3730-4028-bc13-0e2e63677856\" (UID: \"20d0ecca-3730-4028-bc13-0e2e63677856\") " Jul 24 18:06:21.183426 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.183376 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d0ecca-3730-4028-bc13-0e2e63677856-openshift-service-ca\") pod \"20d0ecca-3730-4028-bc13-0e2e63677856\" (UID: \"20d0ecca-3730-4028-bc13-0e2e63677856\") " Jul 24 18:06:21.183673 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.183491 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f" (UID: "be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:06:21.183940 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.183886 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7028d9-1dd2-47b3-b99f-890f3a1e7755-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e7028d9-1dd2-47b3-b99f-890f3a1e7755" (UID: "3e7028d9-1dd2-47b3-b99f-890f3a1e7755"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:06:21.184029 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.183934 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d0ecca-3730-4028-bc13-0e2e63677856-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20d0ecca-3730-4028-bc13-0e2e63677856" (UID: "20d0ecca-3730-4028-bc13-0e2e63677856"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:06:21.185197 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.185176 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7028d9-1dd2-47b3-b99f-890f3a1e7755-kube-api-access-2n42h" (OuterVolumeSpecName: "kube-api-access-2n42h") pod "3e7028d9-1dd2-47b3-b99f-890f3a1e7755" (UID: "3e7028d9-1dd2-47b3-b99f-890f3a1e7755"). InnerVolumeSpecName "kube-api-access-2n42h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:06:21.185393 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.185371 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d0ecca-3730-4028-bc13-0e2e63677856-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20d0ecca-3730-4028-bc13-0e2e63677856" (UID: "20d0ecca-3730-4028-bc13-0e2e63677856"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:06:21.185687 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.185664 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f-kube-api-access-vggdq" (OuterVolumeSpecName: "kube-api-access-vggdq") pod "be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f" (UID: "be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f"). InnerVolumeSpecName "kube-api-access-vggdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:06:21.185687 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.185669 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d0ecca-3730-4028-bc13-0e2e63677856-kube-api-access-dw8wk" (OuterVolumeSpecName: "kube-api-access-dw8wk") pod "20d0ecca-3730-4028-bc13-0e2e63677856" (UID: "20d0ecca-3730-4028-bc13-0e2e63677856"). InnerVolumeSpecName "kube-api-access-dw8wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:06:21.284741 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.284715 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw8wk\" (UniqueName: \"kubernetes.io/projected/20d0ecca-3730-4028-bc13-0e2e63677856-kube-api-access-dw8wk\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 18:06:21.284741 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.284738 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7028d9-1dd2-47b3-b99f-890f3a1e7755-plugins-config-volume\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 18:06:21.284741 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.284749 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20d0ecca-3730-4028-bc13-0e2e63677856-maas-api-tls\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 18:06:21.284980 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.284757 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d0ecca-3730-4028-bc13-0e2e63677856-openshift-service-ca\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 18:06:21.284980 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.284765 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vggdq\" (UniqueName: \"kubernetes.io/projected/be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f-kube-api-access-vggdq\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 18:06:21.284980 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.284773 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n42h\" (UniqueName: \"kubernetes.io/projected/3e7028d9-1dd2-47b3-b99f-890f3a1e7755-kube-api-access-2n42h\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 18:06:21.284980 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.284782 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f-plugins-config-volume\") on node \"ip-10-0-137-105.ec2.internal\" DevicePath \"\"" Jul 24 18:06:21.839704 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.839664 2559 generic.go:356] "Generic (PLEG): container finished" podID="3e7028d9-1dd2-47b3-b99f-890f3a1e7755" containerID="3a5fd46ed8564f5fd44b4849f72ac45971c35bb38bdb38e7c570ef7c28a9962c" exitCode=0 Jul 24 18:06:21.840128 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.839711 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" Jul 24 18:06:21.840128 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.839770 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" event={"ID":"3e7028d9-1dd2-47b3-b99f-890f3a1e7755","Type":"ContainerDied","Data":"3a5fd46ed8564f5fd44b4849f72ac45971c35bb38bdb38e7c570ef7c28a9962c"} Jul 24 18:06:21.840128 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.839820 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54" event={"ID":"3e7028d9-1dd2-47b3-b99f-890f3a1e7755","Type":"ContainerDied","Data":"5af96fbd0d0f2984df8bf29462db4906df5bb46b6683938f97fc265be1fe1c8b"} Jul 24 18:06:21.840128 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.839847 2559 scope.go:117] "RemoveContainer" containerID="3a5fd46ed8564f5fd44b4849f72ac45971c35bb38bdb38e7c570ef7c28a9962c" Jul 24 18:06:21.841077 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.841056 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" event={"ID":"20d0ecca-3730-4028-bc13-0e2e63677856","Type":"ContainerDied","Data":"bf38f80585fca49cf7586228d04eb18368b515708a8e804ede5fa55b67a6b26e"} Jul 24 18:06:21.841140 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.841069 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn" Jul 24 18:06:21.842484 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.842450 2559 generic.go:356] "Generic (PLEG): container finished" podID="be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f" containerID="58efa987cb206f75e2389df4d5d6e1d37d19d35517239f9136c01282e2212018" exitCode=0 Jul 24 18:06:21.842591 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.842495 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" Jul 24 18:06:21.842591 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.842539 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" event={"ID":"be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f","Type":"ContainerDied","Data":"58efa987cb206f75e2389df4d5d6e1d37d19d35517239f9136c01282e2212018"} Jul 24 18:06:21.842591 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.842585 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc" event={"ID":"be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f","Type":"ContainerDied","Data":"b4368d484cd1e7263b358684c24dcc18b0d5e63b786b58b41b623edb494ee9c5"} Jul 24 18:06:21.850663 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.850638 2559 scope.go:117] "RemoveContainer" containerID="3a5fd46ed8564f5fd44b4849f72ac45971c35bb38bdb38e7c570ef7c28a9962c" Jul 24 18:06:21.850961 ip-10-0-137-105 kubenswrapper[2559]: E0724 18:06:21.850937 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5fd46ed8564f5fd44b4849f72ac45971c35bb38bdb38e7c570ef7c28a9962c\": container with ID starting with 3a5fd46ed8564f5fd44b4849f72ac45971c35bb38bdb38e7c570ef7c28a9962c not found: ID does not exist" containerID="3a5fd46ed8564f5fd44b4849f72ac45971c35bb38bdb38e7c570ef7c28a9962c" Jul 24 18:06:21.851027 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.850976 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5fd46ed8564f5fd44b4849f72ac45971c35bb38bdb38e7c570ef7c28a9962c"} err="failed to get container status \"3a5fd46ed8564f5fd44b4849f72ac45971c35bb38bdb38e7c570ef7c28a9962c\": rpc error: code = NotFound desc = could not find container \"3a5fd46ed8564f5fd44b4849f72ac45971c35bb38bdb38e7c570ef7c28a9962c\": container with ID starting with 3a5fd46ed8564f5fd44b4849f72ac45971c35bb38bdb38e7c570ef7c28a9962c not found: ID does not exist" Jul 24 18:06:21.851027 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.851000 2559 scope.go:117] "RemoveContainer" containerID="348c7336f6be5bd7c59e0fa8009c7a5b61055a6c86d801bb19cbd31e60a8b76f" Jul 24 18:06:21.858858 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.858842 2559 scope.go:117] "RemoveContainer" containerID="58efa987cb206f75e2389df4d5d6e1d37d19d35517239f9136c01282e2212018" Jul 24 18:06:21.865917 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.865902 2559 scope.go:117] "RemoveContainer" containerID="58efa987cb206f75e2389df4d5d6e1d37d19d35517239f9136c01282e2212018" Jul 24 18:06:21.866137 ip-10-0-137-105 kubenswrapper[2559]: E0724 18:06:21.866119 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58efa987cb206f75e2389df4d5d6e1d37d19d35517239f9136c01282e2212018\": container with ID starting with 58efa987cb206f75e2389df4d5d6e1d37d19d35517239f9136c01282e2212018 not found: ID does not exist" containerID="58efa987cb206f75e2389df4d5d6e1d37d19d35517239f9136c01282e2212018" Jul 24 18:06:21.866180 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.866146 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58efa987cb206f75e2389df4d5d6e1d37d19d35517239f9136c01282e2212018"} err="failed to get container status \"58efa987cb206f75e2389df4d5d6e1d37d19d35517239f9136c01282e2212018\": rpc error: code = NotFound desc = could not find container \"58efa987cb206f75e2389df4d5d6e1d37d19d35517239f9136c01282e2212018\": container with ID starting with 58efa987cb206f75e2389df4d5d6e1d37d19d35517239f9136c01282e2212018 not found: ID does not exist" Jul 24 18:06:21.869976 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.869955 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn"] Jul 24 18:06:21.877618 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.877590 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-842768d6-67b44d5446-zlpbn"] Jul 24 18:06:21.891667 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.891648 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54"] Jul 24 18:06:21.895786 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.895766 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54"] Jul 24 18:06:21.909138 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.909113 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc"] Jul 24 18:06:21.916744 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:21.916724 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc"] Jul 24 18:06:22.948131 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:22.948036 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20d0ecca-3730-4028-bc13-0e2e63677856" path="/var/lib/kubelet/pods/20d0ecca-3730-4028-bc13-0e2e63677856/volumes" Jul 24 18:06:22.948592 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:22.948564 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e7028d9-1dd2-47b3-b99f-890f3a1e7755" path="/var/lib/kubelet/pods/3e7028d9-1dd2-47b3-b99f-890f3a1e7755/volumes" Jul 24 18:06:22.948941 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:22.948922 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f" path="/var/lib/kubelet/pods/be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f/volumes" Jul 24 18:06:31.553807 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:31.553777 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56bbcff6f4-x6jkx_0aa8b217-bab8-4bee-9a0d-746adb489663/manager/0.log" Jul 24 18:06:32.025789 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:32.025764 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb_4b19c3e1-fd2f-4538-acf1-8013eec8ed9b/copy-manifests/0.log" Jul 24 18:06:32.054335 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:32.054309 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb_4b19c3e1-fd2f-4538-acf1-8013eec8ed9b/manager/0.log" Jul 24 18:06:32.540359 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:32.540330 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8b488659f-z6l8q_c1d01e53-aae5-42eb-ba9b-bd275895eef1/maas-api/0.log" Jul 24 18:06:32.672262 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:32.672232 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748570-5tbbb_c6383c10-4603-47bd-8e4d-9a64a651b153/cleanup/0.log" Jul 24 18:06:33.010098 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:33.010068 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-xvxlk_2943ea2f-1547-4a19-8a27-285093d45c82/postgres/0.log" Jul 24 18:06:35.863391 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:35.863361 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bd989cf94-wrrd6_1ba67174-6d4c-4e6d-91a7-2d64fffb91cf/kube-auth-proxy/0.log" Jul 24 18:06:36.095534 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:36.095484 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-7mkdk_d4402045-7fac-42d2-93be-8ccedb3d0db6/payload-pre-processing/0.log" Jul 24 18:06:36.227840 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:36.227766 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gq4hn_e972f6f7-21a4-4495-b8a5-430a74b986c0/payload-processing/0.log" Jul 24 18:06:36.698324 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:36.698297 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-wccvb_69ae1a7b-7ac0-4852-8299-520ccb85381b/main/0.log" Jul 24 18:06:37.026305 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:37.026278 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4wppf_18a3ca9f-627d-4374-b1db-8658bfde5557/main/0.log" Jul 24 18:06:37.251888 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:37.251862 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-w9kls_62a7c06f-37aa-42ed-9180-ed3f60ceebdd/main/0.log" Jul 24 18:06:44.998506 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:44.998475 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7bpxv_5f6e8751-f60c-4f88-a369-9158d269f5ff/global-pull-secret-syncer/0.log" Jul 24 18:06:45.202708 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:45.202674 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d9bmx_e621415d-ea2d-4520-8353-752e48ab5b39/konnectivity-agent/0.log" Jul 24 18:06:45.297753 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:45.297713 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-105.ec2.internal_a4b95910892a229c48b40c8bd7c3819e/haproxy/0.log" Jul 24 18:06:52.891569 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:52.891544 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-tddr4_f4d212e4-5203-4465-b795-1efa606b69c2/monitoring-plugin/0.log" Jul 24 18:06:53.032658 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:53.032628 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dpj5x_6c214fce-b532-467e-9ece-94121dee447b/node-exporter/0.log" Jul 24 18:06:53.056868 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:53.056836 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dpj5x_6c214fce-b532-467e-9ece-94121dee447b/kube-rbac-proxy/0.log" Jul 24 18:06:53.084048 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:53.083973 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dpj5x_6c214fce-b532-467e-9ece-94121dee447b/init-textfile/0.log" Jul 24 18:06:53.300461 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:53.300435 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e3734a0-d1b3-4f6a-972c-91257d67d902/prometheus/0.log" Jul 24 18:06:53.324858 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:53.324829 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e3734a0-d1b3-4f6a-972c-91257d67d902/config-reloader/0.log" Jul 24 18:06:53.367350 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:53.367266 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e3734a0-d1b3-4f6a-972c-91257d67d902/thanos-sidecar/0.log" Jul 24 18:06:53.405420 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:53.405392 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e3734a0-d1b3-4f6a-972c-91257d67d902/kube-rbac-proxy-web/0.log" Jul 24 18:06:53.436153 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:53.436128 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e3734a0-d1b3-4f6a-972c-91257d67d902/kube-rbac-proxy/0.log" Jul 24 18:06:53.464602 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:53.464574 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e3734a0-d1b3-4f6a-972c-91257d67d902/kube-rbac-proxy-thanos/0.log" Jul 24 18:06:53.495885 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:53.495864 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e3734a0-d1b3-4f6a-972c-91257d67d902/init-config-reloader/0.log" Jul 24 18:06:54.826560 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.826506 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj"] Jul 24 18:06:54.826938 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.826842 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f" containerName="payload-processing" Jul 24 18:06:54.826938 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.826853 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f" containerName="payload-processing" Jul 24 18:06:54.826938 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.826864 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7028d9-1dd2-47b3-b99f-890f3a1e7755" containerName="payload-pre-processing" Jul 24 18:06:54.826938 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.826869 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7028d9-1dd2-47b3-b99f-890f3a1e7755" containerName="payload-pre-processing" Jul 24 18:06:54.826938 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.826876 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d0ecca-3730-4028-bc13-0e2e63677856" containerName="maas-api" Jul 24 18:06:54.826938 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.826883 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d0ecca-3730-4028-bc13-0e2e63677856" containerName="maas-api" Jul 24 18:06:54.827125 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.826952 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e7028d9-1dd2-47b3-b99f-890f3a1e7755" containerName="payload-pre-processing" Jul 24 18:06:54.827125 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.826960 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d0ecca-3730-4028-bc13-0e2e63677856" containerName="maas-api" Jul 24 18:06:54.827125 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.826969 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="be2ee6c7-7a33-4d5e-b9bb-1df9cec7be6f" containerName="payload-processing" Jul 24 18:06:54.830082 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.830061 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.832803 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.832778 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5wz2s\"/\"default-dockercfg-85krf\"" Jul 24 18:06:54.832920 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.832841 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5wz2s\"/\"openshift-service-ca.crt\"" Jul 24 18:06:54.833853 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.833835 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5wz2s\"/\"kube-root-ca.crt\"" Jul 24 18:06:54.839170 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.839149 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj"] Jul 24 18:06:54.879628 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.879586 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-proc\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.879628 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.879628 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnx85\" (UniqueName: \"kubernetes.io/projected/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-kube-api-access-vnx85\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.879899 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.879767 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-lib-modules\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.879899 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.879810 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-podres\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.879899 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.879870 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-sys\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.980596 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.980563 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-lib-modules\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.980596 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.980602 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-podres\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.980824 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.980630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-sys\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.980824 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.980663 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-proc\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.980824 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.980699 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnx85\" (UniqueName: \"kubernetes.io/projected/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-kube-api-access-vnx85\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.980824 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.980752 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-sys\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.980824 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.980754 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-podres\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.980824 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.980754 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-proc\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.980824 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.980754 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-lib-modules\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:54.989994 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:54.989967 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnx85\" (UniqueName: \"kubernetes.io/projected/9fc0a739-4419-4cfc-9ab3-8b23d2601f03-kube-api-access-vnx85\") pod \"perf-node-gather-daemonset-b62tj\" (UID: \"9fc0a739-4419-4cfc-9ab3-8b23d2601f03\") " pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:55.016367 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:55.016341 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-t9lpl_1b24b7a7-177c-450f-84df-9d8db31c552f/networking-console-plugin/0.log" Jul 24 18:06:55.140827 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:55.140724 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:55.279398 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:55.279372 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj"] Jul 24 18:06:55.281494 ip-10-0-137-105 kubenswrapper[2559]: W0724 18:06:55.281466 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9fc0a739_4419_4cfc_9ab3_8b23d2601f03.slice/crio-51a0da3b94c6027a0265be2b4d75342d2faecce619d49bafee17e0292da91417 WatchSource:0}: Error finding container 51a0da3b94c6027a0265be2b4d75342d2faecce619d49bafee17e0292da91417: Status 404 returned error can't find the container with id 51a0da3b94c6027a0265be2b4d75342d2faecce619d49bafee17e0292da91417 Jul 24 18:06:55.999319 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:55.999280 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" event={"ID":"9fc0a739-4419-4cfc-9ab3-8b23d2601f03","Type":"ContainerStarted","Data":"7b6af004ea4c6fd8bb57a651ccad582a6982fde2af18c3a1f3e05024079a8b6d"} Jul 24 18:06:55.999752 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:55.999431 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:06:55.999752 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:55.999467 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" event={"ID":"9fc0a739-4419-4cfc-9ab3-8b23d2601f03","Type":"ContainerStarted","Data":"51a0da3b94c6027a0265be2b4d75342d2faecce619d49bafee17e0292da91417"} Jul 24 18:06:56.021820 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:56.021763 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" podStartSLOduration=2.021746808 podStartE2EDuration="2.021746808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:06:56.020014685 +0000 UTC m=+2813.671906172" watchObservedRunningTime="2026-07-24 18:06:56.021746808 +0000 UTC m=+2813.673638281" Jul 24 18:06:58.100958 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:58.100929 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wxkxq_f4f49740-2957-426b-9a28-a1f915692dd4/dns/0.log" Jul 24 18:06:58.126756 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:58.126732 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wxkxq_f4f49740-2957-426b-9a28-a1f915692dd4/kube-rbac-proxy/0.log" Jul 24 18:06:58.180927 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:58.180899 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ds7bq_dd6e0d2b-2415-4e7f-b1c8-c3505f80fe96/dns-node-resolver/0.log" Jul 24 18:06:58.805174 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:06:58.805152 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-499zx_7a498a6c-63f1-47ec-a1b5-f04d1b2230ff/node-ca/0.log" Jul 24 18:07:00.077556 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:00.077501 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bd989cf94-wrrd6_1ba67174-6d4c-4e6d-91a7-2d64fffb91cf/kube-auth-proxy/0.log" Jul 24 18:07:00.173488 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:00.173458 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-7mkdk_d4402045-7fac-42d2-93be-8ccedb3d0db6/payload-pre-processing/0.log" Jul 24 18:07:00.206152 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:00.206130 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gq4hn_e972f6f7-21a4-4495-b8a5-430a74b986c0/payload-processing/0.log" Jul 24 18:07:00.958964 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:00.958941 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5fcw2_65334c29-2f2c-4b2f-8e30-acf551b4355a/serve-healthcheck-canary/0.log" Jul 24 18:07:01.744202 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:01.744174 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ptzgn_c7ca7e6f-0549-47c4-ba36-7195702ca2fd/kube-rbac-proxy/0.log" Jul 24 18:07:01.769564 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:01.769535 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ptzgn_c7ca7e6f-0549-47c4-ba36-7195702ca2fd/exporter/0.log" Jul 24 18:07:01.804804 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:01.804775 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ptzgn_c7ca7e6f-0549-47c4-ba36-7195702ca2fd/extractor/0.log" Jul 24 18:07:02.013927 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:02.013895 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5wz2s/perf-node-gather-daemonset-b62tj" Jul 24 18:07:04.558163 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:04.558135 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56bbcff6f4-x6jkx_0aa8b217-bab8-4bee-9a0d-746adb489663/manager/0.log" Jul 24 18:07:04.875219 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:04.875189 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb_4b19c3e1-fd2f-4538-acf1-8013eec8ed9b/manager/0.log" Jul 24 18:07:04.925354 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:04.925330 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dc5bcb4dc-dcmfb_4b19c3e1-fd2f-4538-acf1-8013eec8ed9b/copy-manifests/0.log" Jul 24 18:07:05.520767 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:05.520743 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bd989cf94-wrrd6_1ba67174-6d4c-4e6d-91a7-2d64fffb91cf/kube-auth-proxy/0.log" Jul 24 18:07:05.614218 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:05.614193 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-7mkdk_d4402045-7fac-42d2-93be-8ccedb3d0db6/payload-pre-processing/0.log" Jul 24 18:07:05.643907 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:05.643881 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gq4hn_e972f6f7-21a4-4495-b8a5-430a74b986c0/payload-processing/0.log" Jul 24 18:07:06.362370 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:06.362344 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-bdcf88f84-zx57k_ca87879c-f84d-42b9-874b-5fe5e3fa3faf/manager/0.log" Jul 24 18:07:07.110457 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:07.110428 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bd989cf94-wrrd6_1ba67174-6d4c-4e6d-91a7-2d64fffb91cf/kube-auth-proxy/0.log" Jul 24 18:07:07.194283 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:07.194235 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-7mkdk_d4402045-7fac-42d2-93be-8ccedb3d0db6/payload-pre-processing/0.log" Jul 24 18:07:07.220881 ip-10-0-137-105 kubenswrapper[2559]: I0724 18:07:07.220861 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gq4hn_e972f6f7-21a4-4495-b8a5-430a74b986c0/payload-processing/0.log"