Jul 21 12:25:28.208645 ip-10-0-138-11 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 21 12:25:28.208658 ip-10-0-138-11 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 21 12:25:28.208670 ip-10-0-138-11 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 21 12:25:28.209014 ip-10-0-138-11 systemd[1]: Failed to start Kubernetes Kubelet. Jul 21 12:25:38.278402 ip-10-0-138-11 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 21 12:25:38.278420 ip-10-0-138-11 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 4679a5641af54330813be42c23a1e8ab -- Jul 21 12:27:31.614485 ip-10-0-138-11 systemd[1]: Starting Kubernetes Kubelet... Jul 21 12:27:32.171205 ip-10-0-138-11 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 12:27:32.171205 ip-10-0-138-11 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 12:27:32.171205 ip-10-0-138-11 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 12:27:32.171205 ip-10-0-138-11 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 12:27:32.171205 ip-10-0-138-11 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 12:27:32.171205 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.063318 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 12:27:32.171205 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067712 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 12:27:32.171205 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067728 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 12:27:32.171205 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067731 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 12:27:32.171205 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067734 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 12:27:32.171205 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067738 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 12:27:32.171205 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067740 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 12:27:32.171205 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067743 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 12:27:32.171205 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067748 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 12:27:32.140082 ip-10-0-138-11 systemd[1]: Started Kubernetes Kubelet. Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067752 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067755 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067760 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067764 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067767 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067770 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067774 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067781 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067784 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067787 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067790 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067793 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067795 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067798 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067800 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067803 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067806 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067808 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 12:27:32.172564 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067811 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067813 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067816 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067818 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067821 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067824 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067827 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067829 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067831 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067834 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067837 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067839 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067842 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067845 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067848 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067852 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067854 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067857 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067859 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 12:27:32.173543 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067861 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067864 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067867 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067870 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067872 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067875 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067878 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067880 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067883 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067886 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067888 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067891 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067893 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067896 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067898 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067902 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067904 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067907 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067909 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067912 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 12:27:32.174528 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067914 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067938 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067941 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067944 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067947 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067951 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067955 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067958 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067961 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067964 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067966 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067969 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067971 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067974 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067977 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067980 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067982 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067985 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067987 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 12:27:32.175351 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067990 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067992 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067995 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.067998 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068000 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068003 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068005 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068008 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068011 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068014 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068016 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068019 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068022 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068025 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068027 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068763 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068769 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068773 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068775 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068778 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 12:27:32.175995 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068781 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068784 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068787 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068790 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068793 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068796 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068798 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068801 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068803 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068807 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068811 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068815 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068819 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068822 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068825 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068827 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068830 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068833 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068835 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 12:27:32.176662 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068838 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068840 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068843 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068845 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068848 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068850 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068853 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068856 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068859 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068861 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068864 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068867 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068870 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068873 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068877 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068882 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068885 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068888 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068890 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 12:27:32.177410 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068893 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068895 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068898 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068901 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068904 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068907 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068909 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068912 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068915 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068917 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068920 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068923 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068925 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068927 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068931 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068933 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068936 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068939 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068941 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068944 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 12:27:32.177994 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068947 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068950 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068955 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068958 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068961 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068964 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068967 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068969 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068972 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068975 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068977 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068980 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068983 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068986 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068989 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068992 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068995 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.068998 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069001 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069004 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 12:27:32.178678 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069006 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069009 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069012 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069014 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069017 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069019 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069022 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069024 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069027 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069030 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069032 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069035 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069037 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069040 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069042 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069045 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069455 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069460 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069463 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069466 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 12:27:32.179272 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069469 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069471 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069474 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069477 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069480 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069482 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069485 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069490 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069494 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069499 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069502 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069505 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069507 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069510 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069513 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069515 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069518 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069525 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069527 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 12:27:32.180442 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069530 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069533 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069535 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069538 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069540 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069543 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069546 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069550 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069554 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069556 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069559 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069561 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069565 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069567 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069570 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069572 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069575 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069578 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069581 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 12:27:32.181029 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069584 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069586 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069589 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069592 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069595 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069598 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069600 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069603 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069606 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069608 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069611 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069615 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069618 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069620 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069623 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069625 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069628 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069630 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069633 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069635 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 12:27:32.182102 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069638 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069640 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069643 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069645 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069648 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069650 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069653 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069655 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069658 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069660 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069663 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069667 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069670 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069672 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069676 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069679 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069681 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069684 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069687 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069689 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 12:27:32.182916 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069692 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069694 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069697 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069702 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069706 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069709 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069712 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069715 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069718 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069720 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069723 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069726 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069728 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069731 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069733 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069736 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.069739 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069810 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069817 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069823 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 12:27:32.184555 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069828 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069832 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069835 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069843 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069848 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069851 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069854 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069858 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069861 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069865 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069868 2574 flags.go:64] FLAG: --cgroup-root="" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069871 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069875 2574 flags.go:64] FLAG: --client-ca-file="" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069877 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069880 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069886 2574 flags.go:64] FLAG: --cluster-domain="" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069889 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069892 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069895 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069899 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069902 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069905 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069909 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 12:27:32.185406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069912 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069915 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069918 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069921 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069924 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069927 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069931 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069934 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069937 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069940 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069943 2574 flags.go:64] FLAG: --enable-server="true" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069946 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069953 2574 flags.go:64] FLAG: --event-burst="100" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069956 2574 flags.go:64] FLAG: --event-qps="50" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069960 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069963 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069966 2574 flags.go:64] FLAG: --eviction-hard="" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069969 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069972 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069975 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069979 2574 flags.go:64] FLAG: --eviction-soft="" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069982 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069985 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069988 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069992 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 12:27:32.186002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069995 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.069998 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070001 2574 flags.go:64] FLAG: --feature-gates="" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070004 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070007 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070011 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070015 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070017 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070020 2574 flags.go:64] FLAG: --help="false" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070023 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-138-11.ec2.internal" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070026 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070030 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070033 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070036 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070040 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070042 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070046 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070048 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070051 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070055 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070059 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070062 2574 flags.go:64] FLAG: --kube-reserved="" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070065 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070067 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 12:27:32.186802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070070 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070073 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070076 2574 flags.go:64] FLAG: --lock-file="" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070079 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070083 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070086 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070092 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070096 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070099 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070102 2574 flags.go:64] FLAG: --logging-format="text" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070104 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070108 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070110 2574 flags.go:64] FLAG: --manifest-url="" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070114 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070118 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070121 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070138 2574 flags.go:64] FLAG: --max-pods="110" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070142 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070145 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070148 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070151 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070154 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070157 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070160 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070167 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 12:27:32.187613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070170 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070173 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070178 2574 flags.go:64] FLAG: --pod-cidr="" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070181 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070187 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070190 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070193 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070196 2574 flags.go:64] FLAG: --port="10250" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070199 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070202 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fc3b4df07527cfd3" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070205 2574 flags.go:64] FLAG: --qos-reserved="" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070208 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070212 2574 flags.go:64] FLAG: --register-node="true" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070215 2574 flags.go:64] FLAG: --register-with-taints="" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070220 2574 flags.go:64] FLAG: --registry-burst="10" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070223 2574 flags.go:64] FLAG: --registry-qps="5" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070226 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070228 2574 flags.go:64] FLAG: --reserved-memory="" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070233 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070236 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070239 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070242 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070244 2574 flags.go:64] FLAG: --runonce="false" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070247 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070250 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 12:27:32.188327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070253 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070256 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070259 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070262 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070266 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070269 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070272 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070275 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070277 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070280 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070285 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070288 2574 flags.go:64] FLAG: --system-cgroups="" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070290 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070296 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070299 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070301 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070306 2574 flags.go:64] FLAG: --tls-min-version="" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070309 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070311 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070314 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070318 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070323 2574 flags.go:64] FLAG: --v="2" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070327 2574 flags.go:64] FLAG: --version="false" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070331 2574 flags.go:64] FLAG: --vmodule="" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070334 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 12:27:32.189069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070338 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070453 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070457 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070460 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070463 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070466 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070468 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070472 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070475 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070477 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070481 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070483 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070486 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070489 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070492 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070494 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070497 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070501 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070504 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070506 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 12:27:32.189893 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070509 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070512 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070514 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070517 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070520 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070522 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070525 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070528 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070534 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070538 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070542 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070546 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070550 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070554 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070557 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070562 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070566 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070572 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070577 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 12:27:33.927724 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070580 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070583 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070586 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070589 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070592 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070594 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070597 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070599 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070602 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070604 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070609 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070611 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070614 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070617 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070620 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070622 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070625 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070627 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070630 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070632 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 12:27:33.944441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070635 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070639 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070643 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070645 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070648 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070650 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070653 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070655 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070658 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070660 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070663 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070665 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070669 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070672 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070674 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070677 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070679 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070682 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070684 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070687 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 12:27:33.945654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070689 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070692 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070696 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070698 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070701 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070705 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070708 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070710 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070713 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070717 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070720 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070723 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070726 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070730 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070733 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070736 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070739 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070741 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 12:27:33.946441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070744 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070746 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.070749 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.070755 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.077946 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.077966 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.077990 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.077997 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078057 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078062 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078065 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078069 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078072 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078075 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078078 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 12:27:33.947659 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078081 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078084 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078086 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078089 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078093 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078096 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078100 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078103 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078106 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078111 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078114 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078117 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078120 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078123 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078144 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078147 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078150 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078153 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078156 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078158 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 12:27:33.948274 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078161 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078164 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078168 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078171 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078173 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078176 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078180 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078183 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078186 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078189 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078193 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078197 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078200 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078202 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078205 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078207 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078210 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078213 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078216 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 12:27:33.949301 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078218 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078221 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078223 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078226 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078228 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078231 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078234 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078238 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078241 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078243 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078246 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078248 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078251 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078253 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078256 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078258 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078261 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078264 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078266 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 12:27:33.949921 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078269 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078272 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078275 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078277 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078280 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078283 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078286 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078288 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078291 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078293 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078296 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078298 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078301 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078303 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078306 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078308 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078311 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078313 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078316 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078318 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 12:27:33.950459 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078322 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078325 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078328 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078330 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078334 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078336 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078339 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078341 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078344 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078346 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078349 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078352 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078354 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078360 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.078365 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 12:27:33.951061 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078483 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078489 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078493 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078497 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078500 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078503 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078506 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078508 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078511 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078513 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078516 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078518 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078521 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078523 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078526 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078528 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078531 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078533 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078539 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078541 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 12:27:33.951479 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078544 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078547 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078549 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078552 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078554 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078557 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078559 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078562 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078566 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078569 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078573 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078576 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078579 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078582 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078584 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078587 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078589 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078592 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078594 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078597 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 12:27:33.952069 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078599 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078601 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078604 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078606 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078609 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078611 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078614 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078616 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078619 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078621 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078625 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078628 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078630 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078633 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078635 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078638 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078641 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078643 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078646 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078648 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 12:27:33.952605 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078651 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078654 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078658 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078660 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078663 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078665 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078668 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078671 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078673 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078676 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078678 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078681 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078684 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078686 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078688 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078692 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078695 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078698 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078701 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 12:27:33.953203 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078704 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078707 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078709 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078713 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078716 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078718 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078721 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078724 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078726 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078728 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078731 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078733 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078736 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078738 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078741 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078745 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078748 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078750 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078753 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 12:27:33.954001 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:32.078755 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 12:27:33.954650 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.078761 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 12:27:33.954650 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.078898 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 12:27:33.954650 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.081020 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 12:27:33.954650 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.082382 2574 server.go:1019] "Starting client certificate rotation" Jul 21 12:27:33.954650 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.082486 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 12:27:33.954650 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.082520 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 12:27:33.954650 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.086923 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 12:27:33.954650 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.089743 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 12:27:33.954650 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.101043 2574 log.go:25] "Validated CRI v1 runtime API" Jul 21 12:27:33.954650 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.108088 2574 log.go:25] "Validated CRI v1 image API" Jul 21 12:27:33.954650 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.108976 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 12:27:33.954650 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.110857 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 12:27:33.955068 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.112199 2574 fs.go:135] Filesystem UUIDs: map[44601b3b-ba7f-4d73-9ff7-486b313b9dda:/dev/nvme0n1p3 59bad6f7-34e7-4bc9-a41d-76a7874bbab1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 21 12:27:33.955068 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.112215 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.116572 2574 manager.go:216] Machine: {Timestamp:2026-07-21 12:27:32.11604867 +0000 UTC m=+0.385272358 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100573 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a08705218b6eade3a2604e54ba6d9 SystemUUID:ec2a0870-5218-b6ea-de3a-2604e54ba6d9 BootID:4679a564-1af5-4330-813b-e42c23a1e8ab Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d2:59:0a:49:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d2:59:0a:49:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:ee:e1:45:78:64 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.116684 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.116767 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.119620 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.119653 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-11.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.120343 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.120356 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.120363 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.121241 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.122624 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.126472 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.126614 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.126653 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.126670 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.126685 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.126697 2574 kubelet.go:398] "Adding apiserver pod source" Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.126707 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 12:27:33.955303 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.127064 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.127078 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.130155 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.131976 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.132013 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.133057 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.133073 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.133082 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.133090 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.133104 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.133112 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.133125 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.133146 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.133156 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.133164 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.133175 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.133182 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.133819 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.133828 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.138546 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-69lmc" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.138690 2574 server.go:1268] "Started kubelet" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:32.138805 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:32.138881 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.138932 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.139008 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-11.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.139017 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.139108 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.140410 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.140699 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.147029 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-69lmc" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:32.147094 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-11.ec2.internal.18c44dc2f8cbac84 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-11.ec2.internal,UID:ip-10-0-138-11.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-11.ec2.internal,},FirstTimestamp:2026-07-21 12:27:32.138650756 +0000 UTC m=+0.407874445,LastTimestamp:2026-07-21 12:27:32.138650756 +0000 UTC m=+0.407874445,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-11.ec2.internal,}" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.148119 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.148153 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:32.148285 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-11.ec2.internal\" not found" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.148849 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.148865 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.148965 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.149000 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.149008 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 21 12:27:33.955908 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:32.151960 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:32.151985 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.154493 2574 factory.go:55] Registering systemd factory Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.154531 2574 factory.go:223] Registration of the systemd container factory successfully Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:32.154608 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.154732 2574 factory.go:153] Registering CRI-O factory Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.154750 2574 factory.go:223] Registration of the crio container factory successfully Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.154806 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.154839 2574 factory.go:103] Registering Raw factory Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.154857 2574 manager.go:1196] Started watching for new ooms in manager Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.155305 2574 manager.go:349] Starting recovery of all containers Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.175961 2574 manager.go:354] Recovery completed Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.180262 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.182307 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasSufficientMemory" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.182340 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.182354 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasSufficientPID" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.182836 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.182846 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.182861 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.182891 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.183597 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasSufficientMemory" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.183628 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.183641 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasSufficientPID" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.184187 2574 policy_none.go:49] "None policy: Start" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.184200 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.184210 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:32.185291 2574 policy_none.go:47] "Start" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:33.868152 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-11.ec2.internal\" not found" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.869673 2574 manager.go:337] "Starting Device Plugin manager" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:33.869704 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.869716 2574 server.go:87] "Starting device plugin registration server" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.870023 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.870039 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.870132 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.870239 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.870248 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:33.870873 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:33.870918 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-11.ec2.internal\" not found" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.916258 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.917565 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.917606 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.917662 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:33.917708 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 12:27:33.957013 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.919955 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.971003 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.972075 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasSufficientMemory" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.972109 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.972122 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasSufficientPID" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.972151 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:33.979747 2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-11.ec2.internal\" not found" node="ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:33.980622 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:33.980649 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-11.ec2.internal\": node \"ip-10-0-138-11.ec2.internal\" not found" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:33.994933 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-11.ec2.internal\" not found" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.018579 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-11.ec2.internal"] Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.018654 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.020247 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasSufficientMemory" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.020284 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.020296 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasSufficientPID" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.021538 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.021666 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.021697 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.022286 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasSufficientMemory" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.022319 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.022329 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasSufficientPID" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.022287 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasSufficientMemory" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.022394 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.022408 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasSufficientPID" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.023578 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.023604 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.024284 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasSufficientMemory" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.024314 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.024326 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeHasSufficientPID" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:34.050674 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-11.ec2.internal\" not found" node="ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:34.063538 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-11.ec2.internal\" not found" node="ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.068950 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d366b34bcabfece323a34ba96ea76ea6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal\" (UID: \"d366b34bcabfece323a34ba96ea76ea6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.068975 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d366b34bcabfece323a34ba96ea76ea6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal\" (UID: \"d366b34bcabfece323a34ba96ea76ea6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.075845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.068991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebf654d695d98b916ddecbfb68313be1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-11.ec2.internal\" (UID: \"ebf654d695d98b916ddecbfb68313be1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.194074 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:34.095609 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-11.ec2.internal\" not found" Jul 21 12:27:34.194074 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.170107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d366b34bcabfece323a34ba96ea76ea6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal\" (UID: \"d366b34bcabfece323a34ba96ea76ea6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.194074 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.170152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d366b34bcabfece323a34ba96ea76ea6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal\" (UID: \"d366b34bcabfece323a34ba96ea76ea6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.194074 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.170170 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebf654d695d98b916ddecbfb68313be1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-11.ec2.internal\" (UID: \"ebf654d695d98b916ddecbfb68313be1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.194074 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.170195 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebf654d695d98b916ddecbfb68313be1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-11.ec2.internal\" (UID: \"ebf654d695d98b916ddecbfb68313be1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.194074 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.170108 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d366b34bcabfece323a34ba96ea76ea6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal\" (UID: \"d366b34bcabfece323a34ba96ea76ea6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.194074 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.170239 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d366b34bcabfece323a34ba96ea76ea6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal\" (UID: \"d366b34bcabfece323a34ba96ea76ea6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.196459 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:34.196231 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-11.ec2.internal\" not found" Jul 21 12:27:34.357594 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:34.297083 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-11.ec2.internal\" not found" Jul 21 12:27:34.357594 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.354250 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.481526 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.366406 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.481526 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:34.398102 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-11.ec2.internal\" not found" Jul 21 12:27:34.575711 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:34.498603 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-11.ec2.internal\" not found" Jul 21 12:27:34.575711 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.572909 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 12:27:34.598779 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:34.598748 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-11.ec2.internal\" not found" Jul 21 12:27:34.637752 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.637675 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 12:27:34.698895 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:34.698860 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-11.ec2.internal\" not found" Jul 21 12:27:34.701012 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.700998 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 12:27:34.701147 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.701129 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 12:27:34.701210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.701166 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 12:27:34.701210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.701167 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 12:27:34.774820 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.774762 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 12:22:33 +0000 UTC" deadline="2028-03-03 02:14:19.476218504 +0000 UTC" Jul 21 12:27:34.774820 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.774807 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14173h46m44.701413953s" Jul 21 12:27:34.799854 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:34.799810 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-11.ec2.internal\" not found" Jul 21 12:27:34.872844 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.872815 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 12:27:34.968266 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.968184 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.975186 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.975158 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 12:27:34.976212 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.976196 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal" Jul 21 12:27:34.986148 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:34.986124 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 12:27:35.237902 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:35.237864 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd366b34bcabfece323a34ba96ea76ea6.slice/crio-1bfbca51e0076f030697936244f0da4cf7b71f6c8a58987393d4117ecfca81b0 WatchSource:0}: Error finding container 1bfbca51e0076f030697936244f0da4cf7b71f6c8a58987393d4117ecfca81b0: Status 404 returned error can't find the container with id 1bfbca51e0076f030697936244f0da4cf7b71f6c8a58987393d4117ecfca81b0 Jul 21 12:27:35.242205 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.242190 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 12:27:35.458654 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:35.458619 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf654d695d98b916ddecbfb68313be1.slice/crio-2c1c44e1d7935ab95287f1bc9454218e0a61a7ce7d3d851d565956dd4f114530 WatchSource:0}: Error finding container 2c1c44e1d7935ab95287f1bc9454218e0a61a7ce7d3d851d565956dd4f114530: Status 404 returned error can't find the container with id 2c1c44e1d7935ab95287f1bc9454218e0a61a7ce7d3d851d565956dd4f114530 Jul 21 12:27:35.544449 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.544364 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 12:27:35.746426 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.746392 2574 apiserver.go:52] "Watching apiserver" Jul 21 12:27:35.754449 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.754418 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 12:27:35.754817 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.754794 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-5v5zb","openshift-multus/multus-additional-cni-plugins-6mhqp","openshift-multus/multus-z6wml","openshift-multus/network-metrics-daemon-vzml4","openshift-network-operator/iptables-alerter-d5dhk","openshift-ovn-kubernetes/ovnkube-node-bhsp9","kube-system/global-pull-secret-syncer-rz4f4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q","openshift-dns/node-resolver-gqmm7","openshift-image-registry/node-ca-lxclp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal","openshift-network-diagnostics/network-check-target-xjxkh","kube-system/konnectivity-agent-x9n6h","kube-system/kube-apiserver-proxy-ip-10-0-138-11.ec2.internal"] Jul 21 12:27:35.756448 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.756429 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.757649 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.757619 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.758717 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.758697 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 12:27:35.758799 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.758705 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-774nl\"" Jul 21 12:27:35.758799 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.758739 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 12:27:35.759088 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.759075 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:35.759157 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:35.759132 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:27:35.759859 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.759836 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 12:27:35.759928 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.759836 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xmszp\"" Jul 21 12:27:35.759928 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.759836 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 12:27:35.760255 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.760240 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:35.760329 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:35.760310 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:27:35.760561 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.760545 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 12:27:35.762169 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.761559 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d5dhk" Jul 21 12:27:35.764190 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.764170 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.764582 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.764566 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 12:27:35.764683 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.764662 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 12:27:35.765051 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.764713 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 12:27:35.765051 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.764830 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8tnml\"" Jul 21 12:27:35.765380 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.765364 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.766690 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.766667 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 12:27:35.766782 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.766754 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.767555 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.767541 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 12:27:35.767856 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.767837 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 12:27:35.767963 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.767874 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vqdx7\"" Jul 21 12:27:35.767963 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.767899 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 12:27:35.768081 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.768018 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 12:27:35.768133 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.768099 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 12:27:35.768184 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.768133 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l4wzx\"" Jul 21 12:27:35.768184 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.768172 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 12:27:35.768284 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.768190 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 12:27:35.768284 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.768140 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 12:27:35.768284 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.768244 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 12:27:35.768421 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.768313 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 12:27:35.768473 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.768440 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gqmm7" Jul 21 12:27:35.769062 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.769020 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b7dcf\"" Jul 21 12:27:35.769062 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.769025 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 12:27:35.769062 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.769036 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 12:27:35.769655 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.769641 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lxclp" Jul 21 12:27:35.770955 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.770909 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 12:27:35.771175 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.771152 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wdqxt\"" Jul 21 12:27:35.771238 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.771153 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 12:27:35.771238 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.771210 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:35.771308 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:35.771260 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:27:35.771732 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.771718 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 12:27:35.771922 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.771905 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 12:27:35.772006 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.771963 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f869m\"" Jul 21 12:27:35.772179 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.772165 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 12:27:35.772813 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.772795 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x9n6h" Jul 21 12:27:35.776149 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.776131 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 12:27:35.776328 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.776313 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g7qtq\"" Jul 21 12:27:35.776391 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.776362 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 12:27:35.779199 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779148 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-hostroot\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.779199 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvpht\" (UniqueName: \"kubernetes.io/projected/09deecb4-7e46-42cf-804d-634310995a12-kube-api-access-mvpht\") pod \"node-resolver-gqmm7\" (UID: \"09deecb4-7e46-42cf-804d-634310995a12\") " pod="openshift-dns/node-resolver-gqmm7" Jul 21 12:27:35.779199 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7tpl\" (UniqueName: \"kubernetes.io/projected/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-kube-api-access-w7tpl\") pod \"network-metrics-daemon-vzml4\" (UID: \"9559d0ad-feb3-4e5b-95e8-97173ecf07eb\") " pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:35.779338 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779216 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.779338 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-log-socket\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.779338 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779289 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.779338 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779330 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/10f19045-4a86-44fa-9e06-60030a2120ca-ovnkube-config\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.779521 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-run-multus-certs\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.779521 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-sys-fs\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.779521 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779447 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgrgk\" (UniqueName: \"kubernetes.io/projected/e2d7a87a-edcf-48a9-a632-0d855c12313a-kube-api-access-qgrgk\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.779521 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5rf2\" (UniqueName: \"kubernetes.io/projected/10f19045-4a86-44fa-9e06-60030a2120ca-kube-api-access-b5rf2\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.779521 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779496 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-var-lib-kubelet\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.779753 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779520 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-socket-dir\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.779753 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4628bba7-db68-40d8-b131-2e568224e779-iptables-alerter-script\") pod \"iptables-alerter-d5dhk\" (UID: \"4628bba7-db68-40d8-b131-2e568224e779\") " pod="openshift-network-operator/iptables-alerter-d5dhk" Jul 21 12:27:35.779753 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-run-openvswitch\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.779753 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779591 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-run-ovn\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.779753 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779615 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-multus-daemon-config\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.779753 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09deecb4-7e46-42cf-804d-634310995a12-tmp-dir\") pod \"node-resolver-gqmm7\" (UID: \"09deecb4-7e46-42cf-804d-634310995a12\") " pod="openshift-dns/node-resolver-gqmm7" Jul 21 12:27:35.779753 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779663 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:35.779753 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-sysctl-d\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.779753 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779727 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-systemd\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.779753 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-systemd-units\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.780210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779776 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcc293a1-e31f-4f03-850c-e36181f02e59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.780210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779808 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-lib-modules\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.780210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779833 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-registration-dir\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.780210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4628bba7-db68-40d8-b131-2e568224e779-host-slash\") pod \"iptables-alerter-d5dhk\" (UID: \"4628bba7-db68-40d8-b131-2e568224e779\") " pod="openshift-network-operator/iptables-alerter-d5dhk" Jul 21 12:27:35.780210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779879 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-run-netns\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.780210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779931 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-run-systemd\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.780210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.779989 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-cni-binary-copy\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.780210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780011 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-multus-conf-dir\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.780210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780045 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmcm7\" (UniqueName: \"kubernetes.io/projected/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-kube-api-access-mmcm7\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.780210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-os-release\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.780210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780089 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-host\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.780210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2d7a87a-edcf-48a9-a632-0d855c12313a-tmp\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.780210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780133 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-cni-netd\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.780210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780157 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-multus-cni-dir\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780206 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkcck\" (UniqueName: \"kubernetes.io/projected/dcc293a1-e31f-4f03-850c-e36181f02e59-kube-api-access-fkcck\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780243 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-etc-openvswitch\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-run-k8s-cni-cncf-io\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcc293a1-e31f-4f03-850c-e36181f02e59-cnibin\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780320 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcc293a1-e31f-4f03-850c-e36181f02e59-cni-binary-copy\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780344 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-device-dir\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780367 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-node-log\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-cni-bin\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780413 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/10f19045-4a86-44fa-9e06-60030a2120ca-env-overrides\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/10f19045-4a86-44fa-9e06-60030a2120ca-ovnkube-script-lib\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780460 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-system-cni-dir\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-etc-kubernetes\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780549 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-tuned\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-var-lib-openvswitch\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780621 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.780688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc293a1-e31f-4f03-850c-e36181f02e59-system-cni-dir\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-kubelet\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780673 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-sys\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780687 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780713 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/10f19045-4a86-44fa-9e06-60030a2120ca-ovn-node-metrics-cert\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780740 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09deecb4-7e46-42cf-804d-634310995a12-hosts-file\") pod \"node-resolver-gqmm7\" (UID: \"09deecb4-7e46-42cf-804d-634310995a12\") " pod="openshift-dns/node-resolver-gqmm7" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780757 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dcc293a1-e31f-4f03-850c-e36181f02e59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780773 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-modprobe-d\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh5nm\" (UniqueName: \"kubernetes.io/projected/36af134d-9531-48fa-8e9d-c8feb3540fe1-kube-api-access-gh5nm\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780807 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-run-netns\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780821 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-kubernetes\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dcc293a1-e31f-4f03-850c-e36181f02e59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780864 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-multus-socket-dir-parent\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780877 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-var-lib-cni-bin\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780894 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-sysconfig\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780914 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-cnibin\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780962 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcc293a1-e31f-4f03-850c-e36181f02e59-os-release\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.781267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780979 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-etc-selinux\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.781754 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.780993 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-sysctl-conf\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.781754 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.781024 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-run\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.781754 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.781045 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-slash\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.781754 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.781064 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-var-lib-cni-multus\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.781754 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.781078 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-kubelet-config\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:35.781754 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.781108 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-var-lib-kubelet\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.781754 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.781142 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsg24\" (UniqueName: \"kubernetes.io/projected/4628bba7-db68-40d8-b131-2e568224e779-kube-api-access-gsg24\") pod \"iptables-alerter-d5dhk\" (UID: \"4628bba7-db68-40d8-b131-2e568224e779\") " pod="openshift-network-operator/iptables-alerter-d5dhk" Jul 21 12:27:35.781754 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.781167 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-dbus\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:35.781754 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.781189 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs\") pod \"network-metrics-daemon-vzml4\" (UID: \"9559d0ad-feb3-4e5b-95e8-97173ecf07eb\") " pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:35.799656 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.799632 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vgjwb" Jul 21 12:27:35.807784 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.807759 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vgjwb" Jul 21 12:27:35.869583 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.869557 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 12:27:35.881600 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-systemd\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.881778 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881608 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-systemd-units\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.881778 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881635 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcc293a1-e31f-4f03-850c-e36181f02e59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.881778 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881681 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-lib-modules\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.881778 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881697 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-registration-dir\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.881778 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881701 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-systemd\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.881778 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4628bba7-db68-40d8-b131-2e568224e779-host-slash\") pod \"iptables-alerter-d5dhk\" (UID: \"4628bba7-db68-40d8-b131-2e568224e779\") " pod="openshift-network-operator/iptables-alerter-d5dhk" Jul 21 12:27:35.881778 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881736 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-run-netns\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.881778 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881761 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-run-systemd\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.881778 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881761 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4628bba7-db68-40d8-b131-2e568224e779-host-slash\") pod \"iptables-alerter-d5dhk\" (UID: \"4628bba7-db68-40d8-b131-2e568224e779\") " pod="openshift-network-operator/iptables-alerter-d5dhk" Jul 21 12:27:35.881778 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881701 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-systemd-units\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.881778 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881774 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-registration-dir\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-cni-binary-copy\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881826 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-multus-conf-dir\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881840 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-run-systemd\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmcm7\" (UniqueName: \"kubernetes.io/projected/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-kube-api-access-mmcm7\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-os-release\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-host\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881926 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2d7a87a-edcf-48a9-a632-0d855c12313a-tmp\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881933 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-lib-modules\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881957 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcc293a1-e31f-4f03-850c-e36181f02e59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-cni-netd\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.881999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-multus-cni-dir\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882004 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-multus-conf-dir\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkcck\" (UniqueName: \"kubernetes.io/projected/dcc293a1-e31f-4f03-850c-e36181f02e59-kube-api-access-fkcck\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882067 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/795fc54b-5129-4ef3-8fba-7aab06517e1f-serviceca\") pod \"node-ca-lxclp\" (UID: \"795fc54b-5129-4ef3-8fba-7aab06517e1f\") " pod="openshift-image-registry/node-ca-lxclp" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882075 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-os-release\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-etc-openvswitch\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882114 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-run-k8s-cni-cncf-io\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.882347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882139 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcc293a1-e31f-4f03-850c-e36181f02e59-cnibin\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcc293a1-e31f-4f03-850c-e36181f02e59-cni-binary-copy\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882182 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-device-dir\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-node-log\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-cni-bin\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882249 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-run-netns\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/10f19045-4a86-44fa-9e06-60030a2120ca-env-overrides\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882288 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-multus-cni-dir\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/10f19045-4a86-44fa-9e06-60030a2120ca-ovnkube-script-lib\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-system-cni-dir\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882375 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-etc-kubernetes\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882397 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-tuned\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-var-lib-openvswitch\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882460 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc293a1-e31f-4f03-850c-e36181f02e59-system-cni-dir\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bcvb\" (UniqueName: \"kubernetes.io/projected/795fc54b-5129-4ef3-8fba-7aab06517e1f-kube-api-access-9bcvb\") pod \"node-ca-lxclp\" (UID: \"795fc54b-5129-4ef3-8fba-7aab06517e1f\") " pod="openshift-image-registry/node-ca-lxclp" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882488 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882551 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-kubelet\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.883182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882598 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-system-cni-dir\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882639 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-etc-kubernetes\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882734 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/10f19045-4a86-44fa-9e06-60030a2120ca-env-overrides\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882745 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-var-lib-openvswitch\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882748 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882778 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc293a1-e31f-4f03-850c-e36181f02e59-system-cni-dir\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882823 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-run-k8s-cni-cncf-io\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882825 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcc293a1-e31f-4f03-850c-e36181f02e59-cnibin\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882855 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-host\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882861 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/10f19045-4a86-44fa-9e06-60030a2120ca-ovnkube-script-lib\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882515 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-kubelet\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882955 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-etc-openvswitch\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-sys\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882965 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-cni-bin\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882994 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-cni-netd\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.882998 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-device-dir\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/10f19045-4a86-44fa-9e06-60030a2120ca-ovn-node-metrics-cert\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883045 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-sys\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.884066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09deecb4-7e46-42cf-804d-634310995a12-hosts-file\") pod \"node-resolver-gqmm7\" (UID: \"09deecb4-7e46-42cf-804d-634310995a12\") " pod="openshift-dns/node-resolver-gqmm7" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dcc293a1-e31f-4f03-850c-e36181f02e59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883125 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-node-log\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883186 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09deecb4-7e46-42cf-804d-634310995a12-hosts-file\") pod \"node-resolver-gqmm7\" (UID: \"09deecb4-7e46-42cf-804d-634310995a12\") " pod="openshift-dns/node-resolver-gqmm7" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883199 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-modprobe-d\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883278 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-cni-binary-copy\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh5nm\" (UniqueName: \"kubernetes.io/projected/36af134d-9531-48fa-8e9d-c8feb3540fe1-kube-api-access-gh5nm\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883336 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-run-netns\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883362 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d20b63ed-d383-4d87-b716-99961b0a4a57-konnectivity-ca\") pod \"konnectivity-agent-x9n6h\" (UID: \"d20b63ed-d383-4d87-b716-99961b0a4a57\") " pod="kube-system/konnectivity-agent-x9n6h" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-kubernetes\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883376 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-modprobe-d\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883401 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcc293a1-e31f-4f03-850c-e36181f02e59-cni-binary-copy\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883460 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-kubernetes\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883498 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-run-netns\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883564 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dcc293a1-e31f-4f03-850c-e36181f02e59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dcc293a1-e31f-4f03-850c-e36181f02e59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-multus-socket-dir-parent\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.884884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-var-lib-cni-bin\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883689 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-sysconfig\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883711 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-cnibin\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883726 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-multus-socket-dir-parent\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883732 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcc293a1-e31f-4f03-850c-e36181f02e59-os-release\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883762 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-etc-selinux\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-sysctl-conf\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-run\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-slash\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883844 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcc293a1-e31f-4f03-850c-e36181f02e59-os-release\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883850 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-var-lib-cni-multus\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883881 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-kubelet-config\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883885 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-var-lib-cni-multus\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883906 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-var-lib-kubelet\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883930 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsg24\" (UniqueName: \"kubernetes.io/projected/4628bba7-db68-40d8-b131-2e568224e779-kube-api-access-gsg24\") pod \"iptables-alerter-d5dhk\" (UID: \"4628bba7-db68-40d8-b131-2e568224e779\") " pod="openshift-network-operator/iptables-alerter-d5dhk" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-dbus\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883980 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-etc-selinux\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.885630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.883999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs\") pod \"network-metrics-daemon-vzml4\" (UID: \"9559d0ad-feb3-4e5b-95e8-97173ecf07eb\") " pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884026 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795fc54b-5129-4ef3-8fba-7aab06517e1f-host\") pod \"node-ca-lxclp\" (UID: \"795fc54b-5129-4ef3-8fba-7aab06517e1f\") " pod="openshift-image-registry/node-ca-lxclp" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884049 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-hostroot\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884070 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-sysctl-conf\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884074 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-run\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884074 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvpht\" (UniqueName: \"kubernetes.io/projected/09deecb4-7e46-42cf-804d-634310995a12-kube-api-access-mvpht\") pod \"node-resolver-gqmm7\" (UID: \"09deecb4-7e46-42cf-804d-634310995a12\") " pod="openshift-dns/node-resolver-gqmm7" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884083 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dcc293a1-e31f-4f03-850c-e36181f02e59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7tpl\" (UniqueName: \"kubernetes.io/projected/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-kube-api-access-w7tpl\") pod \"network-metrics-daemon-vzml4\" (UID: \"9559d0ad-feb3-4e5b-95e8-97173ecf07eb\") " pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884153 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884160 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-kubelet-config\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884153 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-slash\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884197 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-dbus\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884205 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-var-lib-kubelet\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:35.884285 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884123 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-var-lib-cni-bin\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884369 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-sysconfig\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:35.884382 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs podName:9559d0ad-feb3-4e5b-95e8-97173ecf07eb nodeName:}" failed. No retries permitted until 2026-07-21 12:27:36.384338526 +0000 UTC m=+3.034751889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs") pod "network-metrics-daemon-vzml4" (UID: "9559d0ad-feb3-4e5b-95e8-97173ecf07eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:27:35.886402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884387 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884161 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-cnibin\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-log-socket\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884449 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-hostroot\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/10f19045-4a86-44fa-9e06-60030a2120ca-ovnkube-config\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884488 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-log-socket\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884499 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-run-multus-certs\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884562 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsrrb\" (UniqueName: \"kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb\") pod \"network-check-target-xjxkh\" (UID: \"c4f3ee5b-f5e6-48eb-8820-2a23cc003565\") " pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884587 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d20b63ed-d383-4d87-b716-99961b0a4a57-agent-certs\") pod \"konnectivity-agent-x9n6h\" (UID: \"d20b63ed-d383-4d87-b716-99961b0a4a57\") " pod="kube-system/konnectivity-agent-x9n6h" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884595 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-run-multus-certs\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884618 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-sys-fs\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884643 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgrgk\" (UniqueName: \"kubernetes.io/projected/e2d7a87a-edcf-48a9-a632-0d855c12313a-kube-api-access-qgrgk\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884664 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5rf2\" (UniqueName: \"kubernetes.io/projected/10f19045-4a86-44fa-9e06-60030a2120ca-kube-api-access-b5rf2\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884685 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-var-lib-kubelet\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884706 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-socket-dir\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.887145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884702 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-sys-fs\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884730 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4628bba7-db68-40d8-b131-2e568224e779-iptables-alerter-script\") pod \"iptables-alerter-d5dhk\" (UID: \"4628bba7-db68-40d8-b131-2e568224e779\") " pod="openshift-network-operator/iptables-alerter-d5dhk" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884752 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-run-openvswitch\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884759 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-host-var-lib-kubelet\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884771 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-run-ovn\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884793 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-multus-daemon-config\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884815 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09deecb4-7e46-42cf-804d-634310995a12-tmp-dir\") pod \"node-resolver-gqmm7\" (UID: \"09deecb4-7e46-42cf-804d-634310995a12\") " pod="openshift-dns/node-resolver-gqmm7" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884862 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-sysctl-d\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884891 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-run-openvswitch\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884897 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/10f19045-4a86-44fa-9e06-60030a2120ca-ovnkube-config\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.884994 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/10f19045-4a86-44fa-9e06-60030a2120ca-run-ovn\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.885053 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36af134d-9531-48fa-8e9d-c8feb3540fe1-socket-dir\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.885138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-sysctl-d\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.885230 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09deecb4-7e46-42cf-804d-634310995a12-tmp-dir\") pod \"node-resolver-gqmm7\" (UID: \"09deecb4-7e46-42cf-804d-634310995a12\") " pod="openshift-dns/node-resolver-gqmm7" Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:35.885303 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:35.885368 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret podName:b0be2da5-e2d6-4bae-bdcc-c65eb877fa03 nodeName:}" failed. No retries permitted until 2026-07-21 12:27:36.385353659 +0000 UTC m=+3.035766998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret") pod "global-pull-secret-syncer-rz4f4" (UID: "b0be2da5-e2d6-4bae-bdcc-c65eb877fa03") : object "kube-system"/"original-pull-secret" not registered Jul 21 12:27:35.887802 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.885453 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-multus-daemon-config\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.888387 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.885643 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4628bba7-db68-40d8-b131-2e568224e779-iptables-alerter-script\") pod \"iptables-alerter-d5dhk\" (UID: \"4628bba7-db68-40d8-b131-2e568224e779\") " pod="openshift-network-operator/iptables-alerter-d5dhk" Jul 21 12:27:35.888387 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.886049 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e2d7a87a-edcf-48a9-a632-0d855c12313a-etc-tuned\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.888387 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.886089 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2d7a87a-edcf-48a9-a632-0d855c12313a-tmp\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.888387 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.886160 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/10f19045-4a86-44fa-9e06-60030a2120ca-ovn-node-metrics-cert\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.894364 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.893997 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmcm7\" (UniqueName: \"kubernetes.io/projected/a3c550a7-e6f4-4269-a4eb-c07415f69fc8-kube-api-access-mmcm7\") pod \"multus-z6wml\" (UID: \"a3c550a7-e6f4-4269-a4eb-c07415f69fc8\") " pod="openshift-multus/multus-z6wml" Jul 21 12:27:35.894561 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.894542 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkcck\" (UniqueName: \"kubernetes.io/projected/dcc293a1-e31f-4f03-850c-e36181f02e59-kube-api-access-fkcck\") pod \"multus-additional-cni-plugins-6mhqp\" (UID: \"dcc293a1-e31f-4f03-850c-e36181f02e59\") " pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:35.895336 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.895312 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsg24\" (UniqueName: \"kubernetes.io/projected/4628bba7-db68-40d8-b131-2e568224e779-kube-api-access-gsg24\") pod \"iptables-alerter-d5dhk\" (UID: \"4628bba7-db68-40d8-b131-2e568224e779\") " pod="openshift-network-operator/iptables-alerter-d5dhk" Jul 21 12:27:35.895953 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.895917 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5rf2\" (UniqueName: \"kubernetes.io/projected/10f19045-4a86-44fa-9e06-60030a2120ca-kube-api-access-b5rf2\") pod \"ovnkube-node-bhsp9\" (UID: \"10f19045-4a86-44fa-9e06-60030a2120ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:35.896305 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.896273 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgrgk\" (UniqueName: \"kubernetes.io/projected/e2d7a87a-edcf-48a9-a632-0d855c12313a-kube-api-access-qgrgk\") pod \"tuned-5v5zb\" (UID: \"e2d7a87a-edcf-48a9-a632-0d855c12313a\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:35.896507 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.896490 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvpht\" (UniqueName: \"kubernetes.io/projected/09deecb4-7e46-42cf-804d-634310995a12-kube-api-access-mvpht\") pod \"node-resolver-gqmm7\" (UID: \"09deecb4-7e46-42cf-804d-634310995a12\") " pod="openshift-dns/node-resolver-gqmm7" Jul 21 12:27:35.896738 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.896718 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh5nm\" (UniqueName: \"kubernetes.io/projected/36af134d-9531-48fa-8e9d-c8feb3540fe1-kube-api-access-gh5nm\") pod \"aws-ebs-csi-driver-node-d9j2q\" (UID: \"36af134d-9531-48fa-8e9d-c8feb3540fe1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:35.897803 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.897783 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7tpl\" (UniqueName: \"kubernetes.io/projected/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-kube-api-access-w7tpl\") pod \"network-metrics-daemon-vzml4\" (UID: \"9559d0ad-feb3-4e5b-95e8-97173ecf07eb\") " pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:35.922022 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.921964 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.ec2.internal" event={"ID":"ebf654d695d98b916ddecbfb68313be1","Type":"ContainerStarted","Data":"2c1c44e1d7935ab95287f1bc9454218e0a61a7ce7d3d851d565956dd4f114530"} Jul 21 12:27:35.923652 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.923165 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal" event={"ID":"d366b34bcabfece323a34ba96ea76ea6","Type":"ContainerStarted","Data":"1bfbca51e0076f030697936244f0da4cf7b71f6c8a58987393d4117ecfca81b0"} Jul 21 12:27:35.985240 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.985195 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bcvb\" (UniqueName: \"kubernetes.io/projected/795fc54b-5129-4ef3-8fba-7aab06517e1f-kube-api-access-9bcvb\") pod \"node-ca-lxclp\" (UID: \"795fc54b-5129-4ef3-8fba-7aab06517e1f\") " pod="openshift-image-registry/node-ca-lxclp" Jul 21 12:27:35.985401 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.985258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d20b63ed-d383-4d87-b716-99961b0a4a57-konnectivity-ca\") pod \"konnectivity-agent-x9n6h\" (UID: \"d20b63ed-d383-4d87-b716-99961b0a4a57\") " pod="kube-system/konnectivity-agent-x9n6h" Jul 21 12:27:35.985401 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.985301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795fc54b-5129-4ef3-8fba-7aab06517e1f-host\") pod \"node-ca-lxclp\" (UID: \"795fc54b-5129-4ef3-8fba-7aab06517e1f\") " pod="openshift-image-registry/node-ca-lxclp" Jul 21 12:27:35.985401 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.985331 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrrb\" (UniqueName: \"kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb\") pod \"network-check-target-xjxkh\" (UID: \"c4f3ee5b-f5e6-48eb-8820-2a23cc003565\") " pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:35.985401 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.985351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d20b63ed-d383-4d87-b716-99961b0a4a57-agent-certs\") pod \"konnectivity-agent-x9n6h\" (UID: \"d20b63ed-d383-4d87-b716-99961b0a4a57\") " pod="kube-system/konnectivity-agent-x9n6h" Jul 21 12:27:35.985401 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.985398 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/795fc54b-5129-4ef3-8fba-7aab06517e1f-serviceca\") pod \"node-ca-lxclp\" (UID: \"795fc54b-5129-4ef3-8fba-7aab06517e1f\") " pod="openshift-image-registry/node-ca-lxclp" Jul 21 12:27:35.985910 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.985888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/795fc54b-5129-4ef3-8fba-7aab06517e1f-serviceca\") pod \"node-ca-lxclp\" (UID: \"795fc54b-5129-4ef3-8fba-7aab06517e1f\") " pod="openshift-image-registry/node-ca-lxclp" Jul 21 12:27:35.986671 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.986642 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795fc54b-5129-4ef3-8fba-7aab06517e1f-host\") pod \"node-ca-lxclp\" (UID: \"795fc54b-5129-4ef3-8fba-7aab06517e1f\") " pod="openshift-image-registry/node-ca-lxclp" Jul 21 12:27:35.986789 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.986691 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d20b63ed-d383-4d87-b716-99961b0a4a57-konnectivity-ca\") pod \"konnectivity-agent-x9n6h\" (UID: \"d20b63ed-d383-4d87-b716-99961b0a4a57\") " pod="kube-system/konnectivity-agent-x9n6h" Jul 21 12:27:35.990131 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.990105 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d20b63ed-d383-4d87-b716-99961b0a4a57-agent-certs\") pod \"konnectivity-agent-x9n6h\" (UID: \"d20b63ed-d383-4d87-b716-99961b0a4a57\") " pod="kube-system/konnectivity-agent-x9n6h" Jul 21 12:27:35.993090 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:35.993061 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 12:27:35.993180 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:35.993097 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 12:27:35.993180 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:35.993112 2574 projected.go:196] Error preparing data for projected volume kube-api-access-fsrrb for pod openshift-network-diagnostics/network-check-target-xjxkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:27:35.993180 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:35.993181 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb podName:c4f3ee5b-f5e6-48eb-8820-2a23cc003565 nodeName:}" failed. No retries permitted until 2026-07-21 12:27:36.493162371 +0000 UTC m=+3.143575726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fsrrb" (UniqueName: "kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb") pod "network-check-target-xjxkh" (UID: "c4f3ee5b-f5e6-48eb-8820-2a23cc003565") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:27:35.995562 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:35.995511 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bcvb\" (UniqueName: \"kubernetes.io/projected/795fc54b-5129-4ef3-8fba-7aab06517e1f-kube-api-access-9bcvb\") pod \"node-ca-lxclp\" (UID: \"795fc54b-5129-4ef3-8fba-7aab06517e1f\") " pod="openshift-image-registry/node-ca-lxclp" Jul 21 12:27:36.068675 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.068599 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" Jul 21 12:27:36.075956 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.075914 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" Jul 21 12:27:36.079281 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:36.079246 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d7a87a_edcf_48a9_a632_0d855c12313a.slice/crio-425dc02718649d5b12a6d0c2867dda2faa382d73865929debcec402dcc30e5f0 WatchSource:0}: Error finding container 425dc02718649d5b12a6d0c2867dda2faa382d73865929debcec402dcc30e5f0: Status 404 returned error can't find the container with id 425dc02718649d5b12a6d0c2867dda2faa382d73865929debcec402dcc30e5f0 Jul 21 12:27:36.084087 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.083894 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d5dhk" Jul 21 12:27:36.087696 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.087674 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:27:36.095706 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.095681 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z6wml" Jul 21 12:27:36.096290 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:36.096265 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4628bba7_db68_40d8_b131_2e568224e779.slice/crio-b157701eda6e350071b8235b4946bda10d6b2e297350e48251394a2e4cec60f6 WatchSource:0}: Error finding container b157701eda6e350071b8235b4946bda10d6b2e297350e48251394a2e4cec60f6: Status 404 returned error can't find the container with id b157701eda6e350071b8235b4946bda10d6b2e297350e48251394a2e4cec60f6 Jul 21 12:27:36.100130 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.099917 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6mhqp" Jul 21 12:27:36.105710 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.105678 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gqmm7" Jul 21 12:27:36.113038 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.112979 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lxclp" Jul 21 12:27:36.117535 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.117515 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x9n6h" Jul 21 12:27:36.146953 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:27:36.146900 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20b63ed_d383_4d87_b716_99961b0a4a57.slice/crio-8023d1329b46f097b7db6e511e944e17fb42790fa20e84c203de9f95b059f0f0 WatchSource:0}: Error finding container 8023d1329b46f097b7db6e511e944e17fb42790fa20e84c203de9f95b059f0f0: Status 404 returned error can't find the container with id 8023d1329b46f097b7db6e511e944e17fb42790fa20e84c203de9f95b059f0f0 Jul 21 12:27:36.387404 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.387318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs\") pod \"network-metrics-daemon-vzml4\" (UID: \"9559d0ad-feb3-4e5b-95e8-97173ecf07eb\") " pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:36.387404 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.387385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:36.387627 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:36.387457 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:27:36.387627 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:36.387494 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 12:27:36.387627 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:36.387525 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs podName:9559d0ad-feb3-4e5b-95e8-97173ecf07eb nodeName:}" failed. No retries permitted until 2026-07-21 12:27:37.387505823 +0000 UTC m=+4.037919183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs") pod "network-metrics-daemon-vzml4" (UID: "9559d0ad-feb3-4e5b-95e8-97173ecf07eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:27:36.387627 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:36.387544 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret podName:b0be2da5-e2d6-4bae-bdcc-c65eb877fa03 nodeName:}" failed. No retries permitted until 2026-07-21 12:27:37.387534491 +0000 UTC m=+4.037947839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret") pod "global-pull-secret-syncer-rz4f4" (UID: "b0be2da5-e2d6-4bae-bdcc-c65eb877fa03") : object "kube-system"/"original-pull-secret" not registered Jul 21 12:27:36.588875 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.588835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrrb\" (UniqueName: \"kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb\") pod \"network-check-target-xjxkh\" (UID: \"c4f3ee5b-f5e6-48eb-8820-2a23cc003565\") " pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:36.589091 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:36.589041 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 12:27:36.589091 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:36.589068 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 12:27:36.589091 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:36.589081 2574 projected.go:196] Error preparing data for projected volume kube-api-access-fsrrb for pod openshift-network-diagnostics/network-check-target-xjxkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:27:36.589226 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:36.589151 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb podName:c4f3ee5b-f5e6-48eb-8820-2a23cc003565 nodeName:}" failed. No retries permitted until 2026-07-21 12:27:37.589132388 +0000 UTC m=+4.239545749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fsrrb" (UniqueName: "kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb") pod "network-check-target-xjxkh" (UID: "c4f3ee5b-f5e6-48eb-8820-2a23cc003565") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:27:36.810775 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.810669 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 12:22:35 +0000 UTC" deadline="2028-01-25 21:01:36.838462131 +0000 UTC" Jul 21 12:27:36.810775 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.810724 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13280h34m0.027744351s" Jul 21 12:27:36.862693 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.862629 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 12:27:36.927259 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.927193 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lxclp" event={"ID":"795fc54b-5129-4ef3-8fba-7aab06517e1f","Type":"ContainerStarted","Data":"68edd323ce04cc9f98c6c5db0439deaa528fd5ebd1760b527b17944b05ab06c5"} Jul 21 12:27:36.928708 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.928671 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gqmm7" event={"ID":"09deecb4-7e46-42cf-804d-634310995a12","Type":"ContainerStarted","Data":"2657a4f94a90dd2abf90b53c90e511ef387d5766c2ab5bb5705d8f9253457da6"} Jul 21 12:27:36.929952 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.929908 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mhqp" event={"ID":"dcc293a1-e31f-4f03-850c-e36181f02e59","Type":"ContainerStarted","Data":"38caee5c6b2eed1fafe112c77a943f22e557b0d00b3079c178db27693d9d97a8"} Jul 21 12:27:36.931309 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.931267 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" event={"ID":"10f19045-4a86-44fa-9e06-60030a2120ca","Type":"ContainerStarted","Data":"d680f614fe0e68671bba078a85f43fece6b4fd79a6e8a162ba99afcc222257a3"} Jul 21 12:27:36.932522 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.932473 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d5dhk" event={"ID":"4628bba7-db68-40d8-b131-2e568224e779","Type":"ContainerStarted","Data":"b157701eda6e350071b8235b4946bda10d6b2e297350e48251394a2e4cec60f6"} Jul 21 12:27:36.933919 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.933884 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" event={"ID":"e2d7a87a-edcf-48a9-a632-0d855c12313a","Type":"ContainerStarted","Data":"425dc02718649d5b12a6d0c2867dda2faa382d73865929debcec402dcc30e5f0"} Jul 21 12:27:36.935073 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.935045 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x9n6h" event={"ID":"d20b63ed-d383-4d87-b716-99961b0a4a57","Type":"ContainerStarted","Data":"8023d1329b46f097b7db6e511e944e17fb42790fa20e84c203de9f95b059f0f0"} Jul 21 12:27:36.936414 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.936342 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z6wml" event={"ID":"a3c550a7-e6f4-4269-a4eb-c07415f69fc8","Type":"ContainerStarted","Data":"4196273df9bcf9041130f7b1150fb75b33171151efb81d37d966a4a0403f919e"} Jul 21 12:27:36.937540 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:36.937512 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" event={"ID":"36af134d-9531-48fa-8e9d-c8feb3540fe1","Type":"ContainerStarted","Data":"8f0b1bbb5ff330492d865f86b093d5cd1e0eebab12fdd5cd5f31959996735c50"} Jul 21 12:27:37.395406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:37.395353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs\") pod \"network-metrics-daemon-vzml4\" (UID: \"9559d0ad-feb3-4e5b-95e8-97173ecf07eb\") " pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:37.395600 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:37.395448 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:37.395669 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:37.395599 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 12:27:37.395669 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:37.395662 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret podName:b0be2da5-e2d6-4bae-bdcc-c65eb877fa03 nodeName:}" failed. No retries permitted until 2026-07-21 12:27:39.395643163 +0000 UTC m=+6.046056504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret") pod "global-pull-secret-syncer-rz4f4" (UID: "b0be2da5-e2d6-4bae-bdcc-c65eb877fa03") : object "kube-system"/"original-pull-secret" not registered Jul 21 12:27:37.396100 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:37.396072 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:27:37.396213 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:37.396144 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs podName:9559d0ad-feb3-4e5b-95e8-97173ecf07eb nodeName:}" failed. No retries permitted until 2026-07-21 12:27:39.3961274 +0000 UTC m=+6.046540759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs") pod "network-metrics-daemon-vzml4" (UID: "9559d0ad-feb3-4e5b-95e8-97173ecf07eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:27:37.513064 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:37.512842 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 12:27:37.596857 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:37.596818 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrrb\" (UniqueName: \"kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb\") pod \"network-check-target-xjxkh\" (UID: \"c4f3ee5b-f5e6-48eb-8820-2a23cc003565\") " pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:37.597084 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:37.597066 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 12:27:37.597158 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:37.597093 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 12:27:37.597158 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:37.597107 2574 projected.go:196] Error preparing data for projected volume kube-api-access-fsrrb for pod openshift-network-diagnostics/network-check-target-xjxkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:27:37.597251 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:37.597170 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb podName:c4f3ee5b-f5e6-48eb-8820-2a23cc003565 nodeName:}" failed. No retries permitted until 2026-07-21 12:27:39.597150586 +0000 UTC m=+6.247563940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fsrrb" (UniqueName: "kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb") pod "network-check-target-xjxkh" (UID: "c4f3ee5b-f5e6-48eb-8820-2a23cc003565") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:27:37.811442 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:37.811405 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 12:22:35 +0000 UTC" deadline="2028-03-21 20:50:58.077972532 +0000 UTC" Jul 21 12:27:37.811442 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:37.811442 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14624h23m20.266535231s" Jul 21 12:27:37.918958 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:37.918914 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:37.919169 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:37.919124 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:27:37.919248 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:37.919231 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:37.919460 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:37.919391 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:37.919522 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:37.919496 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:27:37.920152 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:37.920119 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:27:37.958622 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:37.957699 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.ec2.internal" event={"ID":"ebf654d695d98b916ddecbfb68313be1","Type":"ContainerStarted","Data":"6b1ec8efe92fca51f655e0d710a11652702629559de18210da13ae30b7bb46da"} Jul 21 12:27:38.969122 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:38.968932 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal" event={"ID":"d366b34bcabfece323a34ba96ea76ea6","Type":"ContainerStarted","Data":"54a5b5cb320972a13c9d6d5ecdad043189c72c839ca1c42297b3db8a968db9f4"} Jul 21 12:27:38.985965 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:38.985055 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.ec2.internal" podStartSLOduration=4.98503181 podStartE2EDuration="4.98503181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:27:37.975459844 +0000 UTC m=+4.625873206" watchObservedRunningTime="2026-07-21 12:27:38.98503181 +0000 UTC m=+5.635445173" Jul 21 12:27:39.415979 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:39.415367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs\") pod \"network-metrics-daemon-vzml4\" (UID: \"9559d0ad-feb3-4e5b-95e8-97173ecf07eb\") " pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:39.415979 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:39.415440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:39.415979 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:39.415566 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 12:27:39.415979 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:39.415631 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret podName:b0be2da5-e2d6-4bae-bdcc-c65eb877fa03 nodeName:}" failed. No retries permitted until 2026-07-21 12:27:43.415611729 +0000 UTC m=+10.066025074 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret") pod "global-pull-secret-syncer-rz4f4" (UID: "b0be2da5-e2d6-4bae-bdcc-c65eb877fa03") : object "kube-system"/"original-pull-secret" not registered Jul 21 12:27:39.415979 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:39.415710 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:27:39.415979 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:39.415773 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs podName:9559d0ad-feb3-4e5b-95e8-97173ecf07eb nodeName:}" failed. No retries permitted until 2026-07-21 12:27:43.415755754 +0000 UTC m=+10.066169117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs") pod "network-metrics-daemon-vzml4" (UID: "9559d0ad-feb3-4e5b-95e8-97173ecf07eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:27:39.617979 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:39.617182 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrrb\" (UniqueName: \"kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb\") pod \"network-check-target-xjxkh\" (UID: \"c4f3ee5b-f5e6-48eb-8820-2a23cc003565\") " pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:39.617979 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:39.617451 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 12:27:39.617979 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:39.617475 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 12:27:39.617979 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:39.617489 2574 projected.go:196] Error preparing data for projected volume kube-api-access-fsrrb for pod openshift-network-diagnostics/network-check-target-xjxkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:27:39.617979 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:39.617570 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb podName:c4f3ee5b-f5e6-48eb-8820-2a23cc003565 nodeName:}" failed. No retries permitted until 2026-07-21 12:27:43.617529934 +0000 UTC m=+10.267943280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fsrrb" (UniqueName: "kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb") pod "network-check-target-xjxkh" (UID: "c4f3ee5b-f5e6-48eb-8820-2a23cc003565") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:27:39.920423 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:39.920347 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:39.920609 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:39.920346 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:39.920609 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:39.920354 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:39.920609 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:39.920523 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:27:39.920766 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:39.920607 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:27:39.920766 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:39.920689 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:27:39.977861 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:39.977056 2574 generic.go:356] "Generic (PLEG): container finished" podID="d366b34bcabfece323a34ba96ea76ea6" containerID="54a5b5cb320972a13c9d6d5ecdad043189c72c839ca1c42297b3db8a968db9f4" exitCode=0 Jul 21 12:27:39.977861 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:39.977110 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal" event={"ID":"d366b34bcabfece323a34ba96ea76ea6","Type":"ContainerDied","Data":"54a5b5cb320972a13c9d6d5ecdad043189c72c839ca1c42297b3db8a968db9f4"} Jul 21 12:27:41.918176 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:41.918141 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:41.918176 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:41.918171 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:41.918718 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:41.918141 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:41.918718 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:41.918310 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:27:41.918718 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:41.918392 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:27:41.918718 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:41.918488 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:27:43.456216 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:43.456170 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs\") pod \"network-metrics-daemon-vzml4\" (UID: \"9559d0ad-feb3-4e5b-95e8-97173ecf07eb\") " pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:43.456682 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:43.456267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:43.456682 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:43.456379 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 12:27:43.456682 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:43.456441 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret podName:b0be2da5-e2d6-4bae-bdcc-c65eb877fa03 nodeName:}" failed. No retries permitted until 2026-07-21 12:27:51.456423318 +0000 UTC m=+18.106836670 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret") pod "global-pull-secret-syncer-rz4f4" (UID: "b0be2da5-e2d6-4bae-bdcc-c65eb877fa03") : object "kube-system"/"original-pull-secret" not registered Jul 21 12:27:43.456682 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:43.456379 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:27:43.456682 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:43.456486 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs podName:9559d0ad-feb3-4e5b-95e8-97173ecf07eb nodeName:}" failed. No retries permitted until 2026-07-21 12:27:51.456476153 +0000 UTC m=+18.106889498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs") pod "network-metrics-daemon-vzml4" (UID: "9559d0ad-feb3-4e5b-95e8-97173ecf07eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:27:43.658369 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:43.658307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrrb\" (UniqueName: \"kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb\") pod \"network-check-target-xjxkh\" (UID: \"c4f3ee5b-f5e6-48eb-8820-2a23cc003565\") " pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:43.658636 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:43.658616 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 12:27:43.658710 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:43.658647 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 12:27:43.658710 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:43.658660 2574 projected.go:196] Error preparing data for projected volume kube-api-access-fsrrb for pod openshift-network-diagnostics/network-check-target-xjxkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:27:43.658818 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:43.658719 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb podName:c4f3ee5b-f5e6-48eb-8820-2a23cc003565 nodeName:}" failed. No retries permitted until 2026-07-21 12:27:51.65870173 +0000 UTC m=+18.309115089 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fsrrb" (UniqueName: "kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb") pod "network-check-target-xjxkh" (UID: "c4f3ee5b-f5e6-48eb-8820-2a23cc003565") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:27:43.918752 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:43.918715 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:43.920342 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:43.919848 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:27:43.920342 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:43.920079 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:43.920342 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:43.920218 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:43.920342 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:43.920213 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:27:43.920342 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:43.920310 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:27:45.918602 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:45.918567 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:45.919111 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:45.918574 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:45.919111 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:45.918713 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:27:45.919111 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:45.918773 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:45.919111 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:45.918869 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:27:45.919111 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:45.918974 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:27:47.918172 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:47.918130 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:47.918644 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:47.918179 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:47.918644 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:47.918269 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:27:47.918644 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:47.918339 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:27:47.918644 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:47.918375 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:47.918644 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:47.918429 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:27:49.918639 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:49.918600 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:49.919096 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:49.918600 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:49.919096 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:49.918738 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:27:49.919096 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:49.918600 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:49.919096 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:49.918812 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:27:49.919096 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:49.918901 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:27:51.518484 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:51.518439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs\") pod \"network-metrics-daemon-vzml4\" (UID: \"9559d0ad-feb3-4e5b-95e8-97173ecf07eb\") " pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:51.519017 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:51.518505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:51.519017 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:51.518601 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:27:51.519017 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:51.518611 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 12:27:51.519017 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:51.518663 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs podName:9559d0ad-feb3-4e5b-95e8-97173ecf07eb nodeName:}" failed. No retries permitted until 2026-07-21 12:28:07.518646941 +0000 UTC m=+34.169060284 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs") pod "network-metrics-daemon-vzml4" (UID: "9559d0ad-feb3-4e5b-95e8-97173ecf07eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:27:51.519017 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:51.518678 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret podName:b0be2da5-e2d6-4bae-bdcc-c65eb877fa03 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:07.518671624 +0000 UTC m=+34.169084962 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret") pod "global-pull-secret-syncer-rz4f4" (UID: "b0be2da5-e2d6-4bae-bdcc-c65eb877fa03") : object "kube-system"/"original-pull-secret" not registered Jul 21 12:27:51.720002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:51.719954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrrb\" (UniqueName: \"kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb\") pod \"network-check-target-xjxkh\" (UID: \"c4f3ee5b-f5e6-48eb-8820-2a23cc003565\") " pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:51.720169 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:51.720123 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 12:27:51.720169 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:51.720146 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 12:27:51.720169 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:51.720157 2574 projected.go:196] Error preparing data for projected volume kube-api-access-fsrrb for pod openshift-network-diagnostics/network-check-target-xjxkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:27:51.720329 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:51.720225 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb podName:c4f3ee5b-f5e6-48eb-8820-2a23cc003565 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:07.720206169 +0000 UTC m=+34.370619508 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fsrrb" (UniqueName: "kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb") pod "network-check-target-xjxkh" (UID: "c4f3ee5b-f5e6-48eb-8820-2a23cc003565") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:27:51.918393 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:51.918311 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:51.918540 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:51.918311 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:51.918540 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:51.918420 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:27:51.918649 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:51.918535 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:27:51.918649 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:51.918311 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:51.918789 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:51.918651 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:27:53.919616 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:53.919575 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:53.920145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:53.919670 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:53.920145 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:53.919701 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:53.920145 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:53.919707 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:27:53.920145 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:53.919778 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:27:53.920145 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:53.919877 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:27:55.918673 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:55.918499 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:55.919210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:55.918499 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:55.919210 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:55.918781 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:27:55.919210 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:55.918845 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:27:55.919210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:55.918544 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:55.919210 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:55.918930 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:27:56.010824 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.010786 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" event={"ID":"e2d7a87a-edcf-48a9-a632-0d855c12313a","Type":"ContainerStarted","Data":"501ece88ac715529bc3f895bbc33aba792c5fde641864adc1255ddb86e68746d"} Jul 21 12:27:56.012256 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.012223 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x9n6h" event={"ID":"d20b63ed-d383-4d87-b716-99961b0a4a57","Type":"ContainerStarted","Data":"f0b00d563f8a573552a18768147ced0a9423a07b071872fed68aa2e6e47677ff"} Jul 21 12:27:56.013733 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.013713 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z6wml" event={"ID":"a3c550a7-e6f4-4269-a4eb-c07415f69fc8","Type":"ContainerStarted","Data":"ab4a99871b9335409b67814901102ea06b2ad8735d8735232972c34539eac34c"} Jul 21 12:27:56.015204 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.015178 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" event={"ID":"36af134d-9531-48fa-8e9d-c8feb3540fe1","Type":"ContainerStarted","Data":"050396bee6d8da70e42bb78ab0b093d15415601b3bcf91ee43fbbf0ee7081e76"} Jul 21 12:27:56.017230 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.017204 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal" event={"ID":"d366b34bcabfece323a34ba96ea76ea6","Type":"ContainerStarted","Data":"580156559a2d4afe26eccc284beb93e1b2b2b67b3da689fee352b27e348c16d1"} Jul 21 12:27:56.018554 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.018533 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lxclp" event={"ID":"795fc54b-5129-4ef3-8fba-7aab06517e1f","Type":"ContainerStarted","Data":"5216647ad0186309d7d62d967be7b5ed5f41d4e6debd1a0bc63196a8d2c58247"} Jul 21 12:27:56.020066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.019998 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gqmm7" event={"ID":"09deecb4-7e46-42cf-804d-634310995a12","Type":"ContainerStarted","Data":"52dd19bf01f03c7a0822f4f541b4eb4f0c7d6660bfed015c4a136b66589fa8b8"} Jul 21 12:27:56.021696 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.021665 2574 generic.go:356] "Generic (PLEG): container finished" podID="dcc293a1-e31f-4f03-850c-e36181f02e59" containerID="69d037a759093f3e6ad76511ab07bc678cf3d9ee7c33c35f5ffe986c2f33b7eb" exitCode=0 Jul 21 12:27:56.021815 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.021736 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mhqp" event={"ID":"dcc293a1-e31f-4f03-850c-e36181f02e59","Type":"ContainerDied","Data":"69d037a759093f3e6ad76511ab07bc678cf3d9ee7c33c35f5ffe986c2f33b7eb"} Jul 21 12:27:56.023251 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.023225 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" event={"ID":"10f19045-4a86-44fa-9e06-60030a2120ca","Type":"ContainerStarted","Data":"474ecb947359900bc15ac509b477d31420d2009b6f412a7e10f168f78265375a"} Jul 21 12:27:56.033158 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.033112 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5v5zb" podStartSLOduration=3.507362965 podStartE2EDuration="23.033097788s" totalImagesPullingTime="19.525734823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:27:56.032770613 +0000 UTC m=+22.683183975" watchObservedRunningTime="2026-07-21 12:27:56.033097788 +0000 UTC m=+22.683511149" Jul 21 12:27:56.047294 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.047234 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gqmm7" podStartSLOduration=2.5585864430000003 podStartE2EDuration="22.047212712s" totalImagesPullingTime="19.488626269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:27:56.046630645 +0000 UTC m=+22.697044006" watchObservedRunningTime="2026-07-21 12:27:56.047212712 +0000 UTC m=+22.697626075" Jul 21 12:27:56.061050 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.061000 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x9n6h" podStartSLOduration=7.401600316 podStartE2EDuration="22.060985248s" totalImagesPullingTime="14.659384932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:27:56.060416703 +0000 UTC m=+22.710830062" watchObservedRunningTime="2026-07-21 12:27:56.060985248 +0000 UTC m=+22.711398608" Jul 21 12:27:56.076632 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.076582 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.ec2.internal" podStartSLOduration=22.076566353 podStartE2EDuration="22.076566353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:27:56.076482184 +0000 UTC m=+22.726895542" watchObservedRunningTime="2026-07-21 12:27:56.076566353 +0000 UTC m=+22.726979714" Jul 21 12:27:56.119344 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.119303 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lxclp" podStartSLOduration=2.643103004 podStartE2EDuration="22.119290827s" totalImagesPullingTime="19.476187823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:27:56.119220102 +0000 UTC m=+22.769633462" watchObservedRunningTime="2026-07-21 12:27:56.119290827 +0000 UTC m=+22.769704188" Jul 21 12:27:56.138909 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:56.138861 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z6wml" podStartSLOduration=2.653751088 podStartE2EDuration="22.138845616s" totalImagesPullingTime="19.485094528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:27:56.138708308 +0000 UTC m=+22.789121671" watchObservedRunningTime="2026-07-21 12:27:56.138845616 +0000 UTC m=+22.789258977" Jul 21 12:27:57.029316 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:57.029046 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:27:57.030175 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:57.029707 2574 generic.go:356] "Generic (PLEG): container finished" podID="10f19045-4a86-44fa-9e06-60030a2120ca" containerID="e396cd252cc9ed1bcc079da97ba33ee096d04d728911f11c84778f5532c63e0b" exitCode=1 Jul 21 12:27:57.030318 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:57.030284 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" event={"ID":"10f19045-4a86-44fa-9e06-60030a2120ca","Type":"ContainerStarted","Data":"ec0bdbf4ea710f91e8587125a64d71dbd0b7ac6d97da4f50d3f4e97b4553a746"} Jul 21 12:27:57.030434 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:57.030406 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" event={"ID":"10f19045-4a86-44fa-9e06-60030a2120ca","Type":"ContainerStarted","Data":"b4517e5c9aec2921fad4a210834636c50c304da9739091bcec847b4d506308f0"} Jul 21 12:27:57.030560 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:57.030460 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" event={"ID":"10f19045-4a86-44fa-9e06-60030a2120ca","Type":"ContainerStarted","Data":"cd5c79632bc02e7031a4b2bc72e8b4d792bd485e1d415f051d0019ce4c9195ce"} Jul 21 12:27:57.030560 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:57.030491 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" event={"ID":"10f19045-4a86-44fa-9e06-60030a2120ca","Type":"ContainerStarted","Data":"2d40189695ef5439ae66bed6226d9b70190e61ab339266b29e3b0bed41d5f40a"} Jul 21 12:27:57.030560 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:57.030518 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" event={"ID":"10f19045-4a86-44fa-9e06-60030a2120ca","Type":"ContainerDied","Data":"e396cd252cc9ed1bcc079da97ba33ee096d04d728911f11c84778f5532c63e0b"} Jul 21 12:27:57.818014 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:57.817981 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 12:27:57.885040 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:57.884904 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T12:27:57.818004349Z","UUID":"62b8f9e9-ada4-43e2-813e-009e65b5ca46","Handler":null,"Name":"","Endpoint":""} Jul 21 12:27:57.887040 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:57.887015 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 12:27:57.887183 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:57.887063 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 12:27:57.918862 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:57.918829 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:57.919054 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:57.918829 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:57.919054 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:57.918972 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:27:57.919054 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:57.919037 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:27:57.919054 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:57.918842 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:57.919281 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:57.919113 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:27:58.034378 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:58.034345 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d5dhk" event={"ID":"4628bba7-db68-40d8-b131-2e568224e779","Type":"ContainerStarted","Data":"ae96bf9b2a15cd672806af252e4d27634d4e281c648af852e53ab2fc598b3518"} Jul 21 12:27:58.036363 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:58.036332 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" event={"ID":"36af134d-9531-48fa-8e9d-c8feb3540fe1","Type":"ContainerStarted","Data":"c5a1a2e175b291241e421f0fbe7c0c4a612f2e05764b69fb20b980e336afdbda"} Jul 21 12:27:58.049045 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:58.049000 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d5dhk" podStartSLOduration=4.560800358 podStartE2EDuration="24.048984829s" totalImagesPullingTime="19.488184471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:27:58.048838676 +0000 UTC m=+24.699252050" watchObservedRunningTime="2026-07-21 12:27:58.048984829 +0000 UTC m=+24.699398189" Jul 21 12:27:59.042107 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:59.042082 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:27:59.042508 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:59.042485 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" event={"ID":"10f19045-4a86-44fa-9e06-60030a2120ca","Type":"ContainerStarted","Data":"ad78bf33f01e4713c25f28dbb696e2c1c1cd1714116dc4d8e987140c0adeea86"} Jul 21 12:27:59.739960 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:59.739759 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x9n6h" Jul 21 12:27:59.740338 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:59.740313 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x9n6h" Jul 21 12:27:59.918191 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:59.918158 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:27:59.918421 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:59.918217 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:27:59.918421 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:27:59.918229 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:27:59.918421 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:59.918334 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:27:59.918589 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:59.918423 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:27:59.918589 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:27:59.918525 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:28:00.046763 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:00.046726 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" event={"ID":"36af134d-9531-48fa-8e9d-c8feb3540fe1","Type":"ContainerStarted","Data":"f64de796fc760e9ed1813ef0ff7f6a39121ab3d1ad2aa5cd638c8edddaafc3af"} Jul 21 12:28:01.051918 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:01.051647 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mhqp" event={"ID":"dcc293a1-e31f-4f03-850c-e36181f02e59","Type":"ContainerStarted","Data":"ff6daa096b2c5f5d3fabab56cc1f3b080d12d0000de255b3ecf695748b24c4d1"} Jul 21 12:28:01.055395 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:01.055374 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:28:01.055782 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:01.055745 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" event={"ID":"10f19045-4a86-44fa-9e06-60030a2120ca","Type":"ContainerStarted","Data":"249ec40b9cb286ed37a2537225d00cec156f1a4967a60425df7309280e15fbc8"} Jul 21 12:28:01.056214 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:01.056200 2574 scope.go:117] "RemoveContainer" containerID="e396cd252cc9ed1bcc079da97ba33ee096d04d728911f11c84778f5532c63e0b" Jul 21 12:28:01.077769 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:01.077738 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x9n6h" Jul 21 12:28:01.077909 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:01.077890 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 21 12:28:01.078413 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:01.078395 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x9n6h" Jul 21 12:28:01.084078 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:01.084033 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9j2q" podStartSLOduration=5.479471125 podStartE2EDuration="28.084015068s" totalImagesPullingTime="22.604543943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:27:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:00.065725697 +0000 UTC m=+26.716139057" watchObservedRunningTime="2026-07-21 12:28:01.084015068 +0000 UTC m=+27.734428430" Jul 21 12:28:01.918516 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:01.918478 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:28:01.918516 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:01.918517 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:28:01.918752 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:01.918490 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:28:01.918752 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:01.918596 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:28:01.918752 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:01.918683 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:28:01.918865 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:01.918758 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:28:02.061099 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:02.061061 2574 generic.go:356] "Generic (PLEG): container finished" podID="dcc293a1-e31f-4f03-850c-e36181f02e59" containerID="ff6daa096b2c5f5d3fabab56cc1f3b080d12d0000de255b3ecf695748b24c4d1" exitCode=0 Jul 21 12:28:02.061681 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:02.061142 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mhqp" event={"ID":"dcc293a1-e31f-4f03-850c-e36181f02e59","Type":"ContainerDied","Data":"ff6daa096b2c5f5d3fabab56cc1f3b080d12d0000de255b3ecf695748b24c4d1"} Jul 21 12:28:02.069887 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:02.069856 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:28:02.073320 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:02.073285 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" event={"ID":"10f19045-4a86-44fa-9e06-60030a2120ca","Type":"ContainerStarted","Data":"14495ec701c782b7f0e831999658a59a434ef7a008fe874cbd4ec9e012dc75c1"} Jul 21 12:28:02.073545 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:02.073525 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:28:02.073636 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:02.073583 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:28:02.073703 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:02.073693 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:28:02.093370 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:02.093334 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:28:02.094103 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:02.094081 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:28:02.119242 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:02.119193 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" podStartSLOduration=8.360970121 podStartE2EDuration="28.119176035s" totalImagesPullingTime="19.758205914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:02.118871691 +0000 UTC m=+28.769285053" watchObservedRunningTime="2026-07-21 12:28:02.119176035 +0000 UTC m=+28.769589395" Jul 21 12:28:03.095330 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:03.095294 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rz4f4"] Jul 21 12:28:03.095803 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:03.095426 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:28:03.095803 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:03.095531 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:28:03.096021 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:03.096001 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xjxkh"] Jul 21 12:28:03.096101 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:03.096090 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:28:03.096172 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:03.096156 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:28:03.097486 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:03.097463 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vzml4"] Jul 21 12:28:03.097596 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:03.097562 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:28:03.097741 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:03.097710 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:28:04.079782 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:04.079584 2574 generic.go:356] "Generic (PLEG): container finished" podID="dcc293a1-e31f-4f03-850c-e36181f02e59" containerID="324f4354e0a9b9dd2e0b07bfb2a9084ea82ba14d1b5847407f378f17a8824ed3" exitCode=0 Jul 21 12:28:04.079994 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:04.079664 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mhqp" event={"ID":"dcc293a1-e31f-4f03-850c-e36181f02e59","Type":"ContainerDied","Data":"324f4354e0a9b9dd2e0b07bfb2a9084ea82ba14d1b5847407f378f17a8824ed3"} Jul 21 12:28:04.917899 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:04.917874 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:28:04.918444 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:04.918000 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:28:04.918444 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:04.918010 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:28:04.918444 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:04.918006 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:28:04.918444 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:04.918210 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:28:04.918444 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:04.918115 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:28:06.086457 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:06.086421 2574 generic.go:356] "Generic (PLEG): container finished" podID="dcc293a1-e31f-4f03-850c-e36181f02e59" containerID="630f5fb55c0372eb3c2dcdee51d077d2b02d95ba14dcdcce100a0e173aeb502a" exitCode=0 Jul 21 12:28:06.087012 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:06.086488 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mhqp" event={"ID":"dcc293a1-e31f-4f03-850c-e36181f02e59","Type":"ContainerDied","Data":"630f5fb55c0372eb3c2dcdee51d077d2b02d95ba14dcdcce100a0e173aeb502a"} Jul 21 12:28:06.918682 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:06.918644 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:28:06.918881 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:06.918644 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:28:06.918881 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:06.918764 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjxkh" podUID="c4f3ee5b-f5e6-48eb-8820-2a23cc003565" Jul 21 12:28:06.918881 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:06.918777 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:28:06.918881 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:06.918869 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzml4" podUID="9559d0ad-feb3-4e5b-95e8-97173ecf07eb" Jul 21 12:28:06.919099 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:06.918985 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz4f4" podUID="b0be2da5-e2d6-4bae-bdcc-c65eb877fa03" Jul 21 12:28:07.539310 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:07.539279 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs\") pod \"network-metrics-daemon-vzml4\" (UID: \"9559d0ad-feb3-4e5b-95e8-97173ecf07eb\") " pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:28:07.539797 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:07.539330 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:28:07.539797 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:07.539447 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 12:28:07.539797 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:07.539451 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:28:07.539797 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:07.539514 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret podName:b0be2da5-e2d6-4bae-bdcc-c65eb877fa03 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:39.539497391 +0000 UTC m=+66.189910732 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret") pod "global-pull-secret-syncer-rz4f4" (UID: "b0be2da5-e2d6-4bae-bdcc-c65eb877fa03") : object "kube-system"/"original-pull-secret" not registered Jul 21 12:28:07.539797 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:07.539531 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs podName:9559d0ad-feb3-4e5b-95e8-97173ecf07eb nodeName:}" failed. No retries permitted until 2026-07-21 12:28:39.539523977 +0000 UTC m=+66.189937315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs") pod "network-metrics-daemon-vzml4" (UID: "9559d0ad-feb3-4e5b-95e8-97173ecf07eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:28:07.740507 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:07.740464 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrrb\" (UniqueName: \"kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb\") pod \"network-check-target-xjxkh\" (UID: \"c4f3ee5b-f5e6-48eb-8820-2a23cc003565\") " pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:28:07.740805 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:07.740625 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 12:28:07.740805 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:07.740643 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 12:28:07.740805 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:07.740651 2574 projected.go:196] Error preparing data for projected volume kube-api-access-fsrrb for pod openshift-network-diagnostics/network-check-target-xjxkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:28:07.740805 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:07.740710 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb podName:c4f3ee5b-f5e6-48eb-8820-2a23cc003565 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:39.740692453 +0000 UTC m=+66.391105795 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fsrrb" (UniqueName: "kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb") pod "network-check-target-xjxkh" (UID: "c4f3ee5b-f5e6-48eb-8820-2a23cc003565") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:28:08.679567 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.679538 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-11.ec2.internal" event="NodeReady" Jul 21 12:28:08.680083 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.679681 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 12:28:08.722528 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.722494 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c4jdq"] Jul 21 12:28:08.725304 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.725279 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:08.725446 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.725414 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t6fkr"] Jul 21 12:28:08.727245 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.727221 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t6fkr" Jul 21 12:28:08.729335 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.729315 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 12:28:08.729335 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.729329 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 12:28:08.729499 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.729339 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r5psm\"" Jul 21 12:28:08.729832 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.729811 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 12:28:08.730078 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.730063 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 12:28:08.730166 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.730104 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6v6zb\"" Jul 21 12:28:08.730166 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.730066 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 12:28:08.735558 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.735535 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c4jdq"] Jul 21 12:28:08.738309 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.738279 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t6fkr"] Jul 21 12:28:08.850243 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.850207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x72tf\" (UniqueName: \"kubernetes.io/projected/9de2e733-a5ab-429f-972e-b375324a5f82-kube-api-access-x72tf\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:08.850450 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.850255 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/493eec89-2b07-4c39-a989-feea5c5a9a1b-cert\") pod \"ingress-canary-t6fkr\" (UID: \"493eec89-2b07-4c39-a989-feea5c5a9a1b\") " pod="openshift-ingress-canary/ingress-canary-t6fkr" Jul 21 12:28:08.850450 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.850385 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9de2e733-a5ab-429f-972e-b375324a5f82-config-volume\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:08.850450 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.850417 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j786x\" (UniqueName: \"kubernetes.io/projected/493eec89-2b07-4c39-a989-feea5c5a9a1b-kube-api-access-j786x\") pod \"ingress-canary-t6fkr\" (UID: \"493eec89-2b07-4c39-a989-feea5c5a9a1b\") " pod="openshift-ingress-canary/ingress-canary-t6fkr" Jul 21 12:28:08.850596 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.850457 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:08.850596 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.850513 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de2e733-a5ab-429f-972e-b375324a5f82-tmp-dir\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:08.918311 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.918270 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:28:08.918476 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.918270 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:28:08.918476 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.918270 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:28:08.921150 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.921096 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 12:28:08.921424 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.921401 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 12:28:08.921546 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.921429 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 12:28:08.921546 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.921455 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 12:28:08.921546 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.921429 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mz2qd\"" Jul 21 12:28:08.921546 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.921491 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-288sp\"" Jul 21 12:28:08.951262 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.951232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9de2e733-a5ab-429f-972e-b375324a5f82-config-volume\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:08.951429 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.951273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j786x\" (UniqueName: \"kubernetes.io/projected/493eec89-2b07-4c39-a989-feea5c5a9a1b-kube-api-access-j786x\") pod \"ingress-canary-t6fkr\" (UID: \"493eec89-2b07-4c39-a989-feea5c5a9a1b\") " pod="openshift-ingress-canary/ingress-canary-t6fkr" Jul 21 12:28:08.951429 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.951399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:08.951539 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.951468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de2e733-a5ab-429f-972e-b375324a5f82-tmp-dir\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:08.951539 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.951500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x72tf\" (UniqueName: \"kubernetes.io/projected/9de2e733-a5ab-429f-972e-b375324a5f82-kube-api-access-x72tf\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:08.951539 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.951527 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/493eec89-2b07-4c39-a989-feea5c5a9a1b-cert\") pod \"ingress-canary-t6fkr\" (UID: \"493eec89-2b07-4c39-a989-feea5c5a9a1b\") " pod="openshift-ingress-canary/ingress-canary-t6fkr" Jul 21 12:28:08.951687 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:08.951551 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 12:28:08.951687 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:08.951622 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls podName:9de2e733-a5ab-429f-972e-b375324a5f82 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:09.451602676 +0000 UTC m=+36.102016030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls") pod "dns-default-c4jdq" (UID: "9de2e733-a5ab-429f-972e-b375324a5f82") : secret "dns-default-metrics-tls" not found Jul 21 12:28:08.951796 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.951773 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de2e733-a5ab-429f-972e-b375324a5f82-tmp-dir\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:08.951922 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.951899 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9de2e733-a5ab-429f-972e-b375324a5f82-config-volume\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:08.956358 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.956335 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/493eec89-2b07-4c39-a989-feea5c5a9a1b-cert\") pod \"ingress-canary-t6fkr\" (UID: \"493eec89-2b07-4c39-a989-feea5c5a9a1b\") " pod="openshift-ingress-canary/ingress-canary-t6fkr" Jul 21 12:28:08.961101 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.961052 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x72tf\" (UniqueName: \"kubernetes.io/projected/9de2e733-a5ab-429f-972e-b375324a5f82-kube-api-access-x72tf\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:08.961244 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:08.961147 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j786x\" (UniqueName: \"kubernetes.io/projected/493eec89-2b07-4c39-a989-feea5c5a9a1b-kube-api-access-j786x\") pod \"ingress-canary-t6fkr\" (UID: \"493eec89-2b07-4c39-a989-feea5c5a9a1b\") " pod="openshift-ingress-canary/ingress-canary-t6fkr" Jul 21 12:28:09.045201 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:09.045165 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t6fkr" Jul 21 12:28:09.454783 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:09.454736 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:09.455001 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:09.454889 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 12:28:09.455001 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:09.454970 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls podName:9de2e733-a5ab-429f-972e-b375324a5f82 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:10.454933133 +0000 UTC m=+37.105346473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls") pod "dns-default-c4jdq" (UID: "9de2e733-a5ab-429f-972e-b375324a5f82") : secret "dns-default-metrics-tls" not found Jul 21 12:28:10.266081 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:10.265781 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t6fkr"] Jul 21 12:28:10.270683 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:28:10.270657 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493eec89_2b07_4c39_a989_feea5c5a9a1b.slice/crio-0e69257845cccc3370f861705f2fb67dace0603a616a4949dc51f8c439264314 WatchSource:0}: Error finding container 0e69257845cccc3370f861705f2fb67dace0603a616a4949dc51f8c439264314: Status 404 returned error can't find the container with id 0e69257845cccc3370f861705f2fb67dace0603a616a4949dc51f8c439264314 Jul 21 12:28:10.461876 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:10.461838 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:10.462049 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:10.462000 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 12:28:10.462097 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:10.462056 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls podName:9de2e733-a5ab-429f-972e-b375324a5f82 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:12.462042315 +0000 UTC m=+39.112455657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls") pod "dns-default-c4jdq" (UID: "9de2e733-a5ab-429f-972e-b375324a5f82") : secret "dns-default-metrics-tls" not found Jul 21 12:28:11.101242 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.101207 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t6fkr" event={"ID":"493eec89-2b07-4c39-a989-feea5c5a9a1b","Type":"ContainerStarted","Data":"0e69257845cccc3370f861705f2fb67dace0603a616a4949dc51f8c439264314"} Jul 21 12:28:11.104406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.104372 2574 generic.go:356] "Generic (PLEG): container finished" podID="dcc293a1-e31f-4f03-850c-e36181f02e59" containerID="39c4196d150a4e3fc7ac2263baf3b18b06ed95b42ba3b2c42a055d8be9baa9a4" exitCode=0 Jul 21 12:28:11.104556 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.104414 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mhqp" event={"ID":"dcc293a1-e31f-4f03-850c-e36181f02e59","Type":"ContainerDied","Data":"39c4196d150a4e3fc7ac2263baf3b18b06ed95b42ba3b2c42a055d8be9baa9a4"} Jul 21 12:28:11.874507 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.874470 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n6plj"] Jul 21 12:28:11.885424 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.885382 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm"] Jul 21 12:28:11.885577 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.885525 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n6plj" Jul 21 12:28:11.889405 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.889357 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 21 12:28:11.889567 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.889425 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 21 12:28:11.889567 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.889357 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-95ctc\"" Jul 21 12:28:11.895754 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.895710 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n6plj"] Jul 21 12:28:11.895754 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.895751 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm"] Jul 21 12:28:11.895925 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.895837 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:11.898829 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.898584 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hdckv\"" Jul 21 12:28:11.898829 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.898670 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 21 12:28:11.898829 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.898760 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 12:28:11.900126 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.900105 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 21 12:28:11.900248 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.900109 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 12:28:11.973336 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.973293 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl67m\" (UniqueName: \"kubernetes.io/projected/4955ba99-c918-417c-8e62-85e8298bf82c-kube-api-access-xl67m\") pod \"volume-data-source-validator-78cf4c6f6-n6plj\" (UID: \"4955ba99-c918-417c-8e62-85e8298bf82c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n6plj" Jul 21 12:28:11.973530 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.973367 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjszb\" (UniqueName: \"kubernetes.io/projected/37341abd-787c-4e7c-a734-514a6a2ebaa0-kube-api-access-sjszb\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:11.973530 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.973459 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:11.973650 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.973598 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/37341abd-787c-4e7c-a734-514a6a2ebaa0-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:11.980258 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.980230 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22"] Jul 21 12:28:11.994103 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.994026 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn"] Jul 21 12:28:11.994256 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.994119 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" Jul 21 12:28:11.998380 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.997419 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nnw4r\"" Jul 21 12:28:11.998380 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.997463 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 21 12:28:11.998380 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.997871 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 21 12:28:11.998380 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.998204 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 21 12:28:11.998380 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:11.998209 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 21 12:28:12.005719 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.005357 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-bc4k4"] Jul 21 12:28:12.006034 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.006009 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" Jul 21 12:28:12.009844 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.009821 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 21 12:28:12.009992 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.009894 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 21 12:28:12.009992 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.009901 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6655v\"" Jul 21 12:28:12.009992 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.009921 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 21 12:28:12.009992 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.009958 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 21 12:28:12.013146 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.013123 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-59v4g"] Jul 21 12:28:12.013319 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.013300 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:12.016049 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.016031 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 21 12:28:12.016241 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.016132 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vjs7q\"" Jul 21 12:28:12.016572 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.016551 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 21 12:28:12.016572 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.016560 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 21 12:28:12.016803 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.016570 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 21 12:28:12.018749 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.018731 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb"] Jul 21 12:28:12.018878 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.018863 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.021723 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.021705 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jt9hz\"" Jul 21 12:28:12.022386 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.022367 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 12:28:12.023039 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.022619 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 12:28:12.023039 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.023023 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 21 12:28:12.023183 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.023032 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 21 12:28:12.024229 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.024209 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 21 12:28:12.025164 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.025087 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" Jul 21 12:28:12.025786 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.025761 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22"] Jul 21 12:28:12.026001 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.025813 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-bc4k4"] Jul 21 12:28:12.026001 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.025838 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn"] Jul 21 12:28:12.026001 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.025857 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb"] Jul 21 12:28:12.026001 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.025884 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-59v4g"] Jul 21 12:28:12.028551 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.028166 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 21 12:28:12.028635 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.028623 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 21 12:28:12.028905 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.028881 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 21 12:28:12.029060 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.029042 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xk9f7\"" Jul 21 12:28:12.033195 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.033179 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 21 12:28:12.074047 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.074006 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8947a59a-697b-41e4-9ecc-dfd60f9eab33-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-w5n22\" (UID: \"8947a59a-697b-41e4-9ecc-dfd60f9eab33\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" Jul 21 12:28:12.074211 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.074061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/37341abd-787c-4e7c-a734-514a6a2ebaa0-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:12.074211 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.074148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl67m\" (UniqueName: \"kubernetes.io/projected/4955ba99-c918-417c-8e62-85e8298bf82c-kube-api-access-xl67m\") pod \"volume-data-source-validator-78cf4c6f6-n6plj\" (UID: \"4955ba99-c918-417c-8e62-85e8298bf82c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n6plj" Jul 21 12:28:12.074211 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.074201 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxhvs\" (UniqueName: \"kubernetes.io/projected/8947a59a-697b-41e4-9ecc-dfd60f9eab33-kube-api-access-cxhvs\") pod \"kube-storage-version-migrator-operator-7658d9b749-w5n22\" (UID: \"8947a59a-697b-41e4-9ecc-dfd60f9eab33\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" Jul 21 12:28:12.074393 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.074244 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8947a59a-697b-41e4-9ecc-dfd60f9eab33-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-w5n22\" (UID: \"8947a59a-697b-41e4-9ecc-dfd60f9eab33\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" Jul 21 12:28:12.074393 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.074278 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjszb\" (UniqueName: \"kubernetes.io/projected/37341abd-787c-4e7c-a734-514a6a2ebaa0-kube-api-access-sjszb\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:12.074393 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.074319 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:12.074538 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:12.074453 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 12:28:12.074594 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:12.074547 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls podName:37341abd-787c-4e7c-a734-514a6a2ebaa0 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:12.574525219 +0000 UTC m=+39.224938563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-qrpqm" (UID: "37341abd-787c-4e7c-a734-514a6a2ebaa0") : secret "cluster-monitoring-operator-tls" not found Jul 21 12:28:12.074914 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.074888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/37341abd-787c-4e7c-a734-514a6a2ebaa0-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:12.087261 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.087227 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl67m\" (UniqueName: \"kubernetes.io/projected/4955ba99-c918-417c-8e62-85e8298bf82c-kube-api-access-xl67m\") pod \"volume-data-source-validator-78cf4c6f6-n6plj\" (UID: \"4955ba99-c918-417c-8e62-85e8298bf82c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n6plj" Jul 21 12:28:12.087421 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.087354 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjszb\" (UniqueName: \"kubernetes.io/projected/37341abd-787c-4e7c-a734-514a6a2ebaa0-kube-api-access-sjszb\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:12.111077 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.111016 2574 generic.go:356] "Generic (PLEG): container finished" podID="dcc293a1-e31f-4f03-850c-e36181f02e59" containerID="6d947cc06501a69029ae59efc51f98f805fa52eef8b027d6b137c6b11b2fcd40" exitCode=0 Jul 21 12:28:12.111077 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.111078 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mhqp" event={"ID":"dcc293a1-e31f-4f03-850c-e36181f02e59","Type":"ContainerDied","Data":"6d947cc06501a69029ae59efc51f98f805fa52eef8b027d6b137c6b11b2fcd40"} Jul 21 12:28:12.175131 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d964404-b5ad-45f6-90c0-e70bb3986442-service-ca-bundle\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.175131 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4d964404-b5ad-45f6-90c0-e70bb3986442-snapshots\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.175131 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175118 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7-serving-cert\") pod \"service-ca-operator-5bcbc94c78-z7shn\" (UID: \"9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" Jul 21 12:28:12.175354 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8947a59a-697b-41e4-9ecc-dfd60f9eab33-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-w5n22\" (UID: \"8947a59a-697b-41e4-9ecc-dfd60f9eab33\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" Jul 21 12:28:12.175354 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175242 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4321016-2119-4390-8984-e74a3d4da956-config\") pod \"console-operator-565b7b569d-bc4k4\" (UID: \"e4321016-2119-4390-8984-e74a3d4da956\") " pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:12.175354 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175279 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxhvs\" (UniqueName: \"kubernetes.io/projected/8947a59a-697b-41e4-9ecc-dfd60f9eab33-kube-api-access-cxhvs\") pod \"kube-storage-version-migrator-operator-7658d9b749-w5n22\" (UID: \"8947a59a-697b-41e4-9ecc-dfd60f9eab33\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" Jul 21 12:28:12.175354 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175300 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4321016-2119-4390-8984-e74a3d4da956-serving-cert\") pod \"console-operator-565b7b569d-bc4k4\" (UID: \"e4321016-2119-4390-8984-e74a3d4da956\") " pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:12.175354 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk9wt\" (UniqueName: \"kubernetes.io/projected/9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7-kube-api-access-sk9wt\") pod \"service-ca-operator-5bcbc94c78-z7shn\" (UID: \"9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" Jul 21 12:28:12.175586 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj47c\" (UniqueName: \"kubernetes.io/projected/b1b704f0-bffb-48d7-bf16-ed9deea793f0-kube-api-access-fj47c\") pod \"cluster-samples-operator-79b7c55f87-b42nb\" (UID: \"b1b704f0-bffb-48d7-bf16-ed9deea793f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" Jul 21 12:28:12.175586 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d964404-b5ad-45f6-90c0-e70bb3986442-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.175586 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4321016-2119-4390-8984-e74a3d4da956-trusted-ca\") pod \"console-operator-565b7b569d-bc4k4\" (UID: \"e4321016-2119-4390-8984-e74a3d4da956\") " pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:12.175724 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175610 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b42nb\" (UID: \"b1b704f0-bffb-48d7-bf16-ed9deea793f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" Jul 21 12:28:12.175724 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175652 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8947a59a-697b-41e4-9ecc-dfd60f9eab33-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-w5n22\" (UID: \"8947a59a-697b-41e4-9ecc-dfd60f9eab33\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" Jul 21 12:28:12.175724 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175695 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7-config\") pod \"service-ca-operator-5bcbc94c78-z7shn\" (UID: \"9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" Jul 21 12:28:12.175869 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpjr6\" (UniqueName: \"kubernetes.io/projected/e4321016-2119-4390-8984-e74a3d4da956-kube-api-access-wpjr6\") pod \"console-operator-565b7b569d-bc4k4\" (UID: \"e4321016-2119-4390-8984-e74a3d4da956\") " pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:12.175869 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175818 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d964404-b5ad-45f6-90c0-e70bb3986442-tmp\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.175988 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175869 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwl96\" (UniqueName: \"kubernetes.io/projected/4d964404-b5ad-45f6-90c0-e70bb3986442-kube-api-access-qwl96\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.175988 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.175922 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d964404-b5ad-45f6-90c0-e70bb3986442-serving-cert\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.176224 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.176204 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8947a59a-697b-41e4-9ecc-dfd60f9eab33-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-w5n22\" (UID: \"8947a59a-697b-41e4-9ecc-dfd60f9eab33\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" Jul 21 12:28:12.178162 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.178138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8947a59a-697b-41e4-9ecc-dfd60f9eab33-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-w5n22\" (UID: \"8947a59a-697b-41e4-9ecc-dfd60f9eab33\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" Jul 21 12:28:12.184637 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.184614 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxhvs\" (UniqueName: \"kubernetes.io/projected/8947a59a-697b-41e4-9ecc-dfd60f9eab33-kube-api-access-cxhvs\") pod \"kube-storage-version-migrator-operator-7658d9b749-w5n22\" (UID: \"8947a59a-697b-41e4-9ecc-dfd60f9eab33\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" Jul 21 12:28:12.197500 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.197471 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n6plj" Jul 21 12:28:12.277031 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.276990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d964404-b5ad-45f6-90c0-e70bb3986442-serving-cert\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.277031 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.277029 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d964404-b5ad-45f6-90c0-e70bb3986442-service-ca-bundle\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.277277 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.277049 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4d964404-b5ad-45f6-90c0-e70bb3986442-snapshots\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.277277 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.277071 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7-serving-cert\") pod \"service-ca-operator-5bcbc94c78-z7shn\" (UID: \"9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" Jul 21 12:28:12.277277 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.277223 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4321016-2119-4390-8984-e74a3d4da956-config\") pod \"console-operator-565b7b569d-bc4k4\" (UID: \"e4321016-2119-4390-8984-e74a3d4da956\") " pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:12.277422 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.277279 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4321016-2119-4390-8984-e74a3d4da956-serving-cert\") pod \"console-operator-565b7b569d-bc4k4\" (UID: \"e4321016-2119-4390-8984-e74a3d4da956\") " pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:12.277422 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.277304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk9wt\" (UniqueName: \"kubernetes.io/projected/9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7-kube-api-access-sk9wt\") pod \"service-ca-operator-5bcbc94c78-z7shn\" (UID: \"9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" Jul 21 12:28:12.277422 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.277340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj47c\" (UniqueName: \"kubernetes.io/projected/b1b704f0-bffb-48d7-bf16-ed9deea793f0-kube-api-access-fj47c\") pod \"cluster-samples-operator-79b7c55f87-b42nb\" (UID: \"b1b704f0-bffb-48d7-bf16-ed9deea793f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" Jul 21 12:28:12.277422 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.277382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d964404-b5ad-45f6-90c0-e70bb3986442-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.277422 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.277416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4321016-2119-4390-8984-e74a3d4da956-trusted-ca\") pod \"console-operator-565b7b569d-bc4k4\" (UID: \"e4321016-2119-4390-8984-e74a3d4da956\") " pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:12.277628 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.277452 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b42nb\" (UID: \"b1b704f0-bffb-48d7-bf16-ed9deea793f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" Jul 21 12:28:12.277628 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:12.277539 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 12:28:12.277628 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:12.277598 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls podName:b1b704f0-bffb-48d7-bf16-ed9deea793f0 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:12.777581385 +0000 UTC m=+39.427994725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b42nb" (UID: "b1b704f0-bffb-48d7-bf16-ed9deea793f0") : secret "samples-operator-tls" not found Jul 21 12:28:12.280247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.278114 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4d964404-b5ad-45f6-90c0-e70bb3986442-snapshots\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.280247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.279062 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d964404-b5ad-45f6-90c0-e70bb3986442-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.280247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.279095 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4321016-2119-4390-8984-e74a3d4da956-config\") pod \"console-operator-565b7b569d-bc4k4\" (UID: \"e4321016-2119-4390-8984-e74a3d4da956\") " pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:12.280247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.279135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7-config\") pod \"service-ca-operator-5bcbc94c78-z7shn\" (UID: \"9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" Jul 21 12:28:12.280247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.279180 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpjr6\" (UniqueName: \"kubernetes.io/projected/e4321016-2119-4390-8984-e74a3d4da956-kube-api-access-wpjr6\") pod \"console-operator-565b7b569d-bc4k4\" (UID: \"e4321016-2119-4390-8984-e74a3d4da956\") " pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:12.280247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.279229 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d964404-b5ad-45f6-90c0-e70bb3986442-tmp\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.280247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.279254 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwl96\" (UniqueName: \"kubernetes.io/projected/4d964404-b5ad-45f6-90c0-e70bb3986442-kube-api-access-qwl96\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.280247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.279798 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7-serving-cert\") pod \"service-ca-operator-5bcbc94c78-z7shn\" (UID: \"9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" Jul 21 12:28:12.280247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.279807 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4321016-2119-4390-8984-e74a3d4da956-trusted-ca\") pod \"console-operator-565b7b569d-bc4k4\" (UID: \"e4321016-2119-4390-8984-e74a3d4da956\") " pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:12.280247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.280052 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d964404-b5ad-45f6-90c0-e70bb3986442-serving-cert\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.280247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.280056 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7-config\") pod \"service-ca-operator-5bcbc94c78-z7shn\" (UID: \"9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" Jul 21 12:28:12.280247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.280078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d964404-b5ad-45f6-90c0-e70bb3986442-tmp\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.280247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.280126 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4321016-2119-4390-8984-e74a3d4da956-serving-cert\") pod \"console-operator-565b7b569d-bc4k4\" (UID: \"e4321016-2119-4390-8984-e74a3d4da956\") " pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:12.280722 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.280473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d964404-b5ad-45f6-90c0-e70bb3986442-service-ca-bundle\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.288534 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.288505 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk9wt\" (UniqueName: \"kubernetes.io/projected/9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7-kube-api-access-sk9wt\") pod \"service-ca-operator-5bcbc94c78-z7shn\" (UID: \"9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" Jul 21 12:28:12.288534 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.288531 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpjr6\" (UniqueName: \"kubernetes.io/projected/e4321016-2119-4390-8984-e74a3d4da956-kube-api-access-wpjr6\") pod \"console-operator-565b7b569d-bc4k4\" (UID: \"e4321016-2119-4390-8984-e74a3d4da956\") " pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:12.288700 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.288529 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwl96\" (UniqueName: \"kubernetes.io/projected/4d964404-b5ad-45f6-90c0-e70bb3986442-kube-api-access-qwl96\") pod \"insights-operator-555d9d7885-59v4g\" (UID: \"4d964404-b5ad-45f6-90c0-e70bb3986442\") " pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.289083 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.289059 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj47c\" (UniqueName: \"kubernetes.io/projected/b1b704f0-bffb-48d7-bf16-ed9deea793f0-kube-api-access-fj47c\") pod \"cluster-samples-operator-79b7c55f87-b42nb\" (UID: \"b1b704f0-bffb-48d7-bf16-ed9deea793f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" Jul 21 12:28:12.307513 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.307475 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" Jul 21 12:28:12.317622 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.317592 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" Jul 21 12:28:12.330761 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.330737 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:12.336296 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.336276 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-59v4g" Jul 21 12:28:12.481835 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.481171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:12.481835 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:12.481412 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 12:28:12.481835 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:12.481480 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls podName:9de2e733-a5ab-429f-972e-b375324a5f82 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:16.481459637 +0000 UTC m=+43.131872979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls") pod "dns-default-c4jdq" (UID: "9de2e733-a5ab-429f-972e-b375324a5f82") : secret "dns-default-metrics-tls" not found Jul 21 12:28:12.584386 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.583658 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:12.584386 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:12.583911 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 12:28:12.584386 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:12.584012 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls podName:37341abd-787c-4e7c-a734-514a6a2ebaa0 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:13.583991815 +0000 UTC m=+40.234405179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-qrpqm" (UID: "37341abd-787c-4e7c-a734-514a6a2ebaa0") : secret "cluster-monitoring-operator-tls" not found Jul 21 12:28:12.738383 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.738132 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn"] Jul 21 12:28:12.750497 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.750469 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22"] Jul 21 12:28:12.753357 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.753329 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-59v4g"] Jul 21 12:28:12.761614 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.761592 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-bc4k4"] Jul 21 12:28:12.770670 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.770645 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n6plj"] Jul 21 12:28:12.785910 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:12.785870 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b42nb\" (UID: \"b1b704f0-bffb-48d7-bf16-ed9deea793f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" Jul 21 12:28:12.786090 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:12.786010 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 12:28:12.786090 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:12.786082 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls podName:b1b704f0-bffb-48d7-bf16-ed9deea793f0 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:13.786061574 +0000 UTC m=+40.436474920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b42nb" (UID: "b1b704f0-bffb-48d7-bf16-ed9deea793f0") : secret "samples-operator-tls" not found Jul 21 12:28:12.894590 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:28:12.894547 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd3ae9a_d294_4c3d_a925_7f3e726f4ee7.slice/crio-427df18929197a63cbd8798701b8312a821f28ec80570618b0a82afeb689ab81 WatchSource:0}: Error finding container 427df18929197a63cbd8798701b8312a821f28ec80570618b0a82afeb689ab81: Status 404 returned error can't find the container with id 427df18929197a63cbd8798701b8312a821f28ec80570618b0a82afeb689ab81 Jul 21 12:28:12.895397 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:28:12.895072 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8947a59a_697b_41e4_9ecc_dfd60f9eab33.slice/crio-d99599fcf3770a1dc4889c0871b211a179b89daa9a9fd1180025a92c03accfea WatchSource:0}: Error finding container d99599fcf3770a1dc4889c0871b211a179b89daa9a9fd1180025a92c03accfea: Status 404 returned error can't find the container with id d99599fcf3770a1dc4889c0871b211a179b89daa9a9fd1180025a92c03accfea Jul 21 12:28:12.895871 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:28:12.895746 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d964404_b5ad_45f6_90c0_e70bb3986442.slice/crio-94e44b4f536b241891c5f8ad8d5e8e6a9825a737fdab96a9830c9bf578700141 WatchSource:0}: Error finding container 94e44b4f536b241891c5f8ad8d5e8e6a9825a737fdab96a9830c9bf578700141: Status 404 returned error can't find the container with id 94e44b4f536b241891c5f8ad8d5e8e6a9825a737fdab96a9830c9bf578700141 Jul 21 12:28:12.896741 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:28:12.896572 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4321016_2119_4390_8984_e74a3d4da956.slice/crio-cc052e1a8a3345e12a85028cf9575bd0ba2ac133b74e7339ea5daa790b64d095 WatchSource:0}: Error finding container cc052e1a8a3345e12a85028cf9575bd0ba2ac133b74e7339ea5daa790b64d095: Status 404 returned error can't find the container with id cc052e1a8a3345e12a85028cf9575bd0ba2ac133b74e7339ea5daa790b64d095 Jul 21 12:28:12.897500 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:28:12.897434 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4955ba99_c918_417c_8e62_85e8298bf82c.slice/crio-9c80458dc9420269492659526bf3bd8cc5b9a1c8dc1fc0d4bd0be7c83416bb06 WatchSource:0}: Error finding container 9c80458dc9420269492659526bf3bd8cc5b9a1c8dc1fc0d4bd0be7c83416bb06: Status 404 returned error can't find the container with id 9c80458dc9420269492659526bf3bd8cc5b9a1c8dc1fc0d4bd0be7c83416bb06 Jul 21 12:28:13.114901 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:13.114862 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-59v4g" event={"ID":"4d964404-b5ad-45f6-90c0-e70bb3986442","Type":"ContainerStarted","Data":"94e44b4f536b241891c5f8ad8d5e8e6a9825a737fdab96a9830c9bf578700141"} Jul 21 12:28:13.120263 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:13.120224 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mhqp" event={"ID":"dcc293a1-e31f-4f03-850c-e36181f02e59","Type":"ContainerStarted","Data":"c76f3974042826a1128f5613b6612413965ee045a702eed813d73c2c74ee6341"} Jul 21 12:28:13.121363 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:13.121337 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" event={"ID":"8947a59a-697b-41e4-9ecc-dfd60f9eab33","Type":"ContainerStarted","Data":"d99599fcf3770a1dc4889c0871b211a179b89daa9a9fd1180025a92c03accfea"} Jul 21 12:28:13.122395 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:13.122370 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" event={"ID":"e4321016-2119-4390-8984-e74a3d4da956","Type":"ContainerStarted","Data":"cc052e1a8a3345e12a85028cf9575bd0ba2ac133b74e7339ea5daa790b64d095"} Jul 21 12:28:13.123430 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:13.123411 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" event={"ID":"9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7","Type":"ContainerStarted","Data":"427df18929197a63cbd8798701b8312a821f28ec80570618b0a82afeb689ab81"} Jul 21 12:28:13.125080 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:13.125056 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t6fkr" event={"ID":"493eec89-2b07-4c39-a989-feea5c5a9a1b","Type":"ContainerStarted","Data":"8e4d5442795ce7dd0357d56d37407ff476dd28e257d3dbd8d7da49680aed52c8"} Jul 21 12:28:13.126171 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:13.126153 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n6plj" event={"ID":"4955ba99-c918-417c-8e62-85e8298bf82c","Type":"ContainerStarted","Data":"9c80458dc9420269492659526bf3bd8cc5b9a1c8dc1fc0d4bd0be7c83416bb06"} Jul 21 12:28:13.144171 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:13.144096 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6mhqp" podStartSLOduration=8.544182675 podStartE2EDuration="39.144075178s" totalImagesPullingTime="30.599892503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:27:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:13.143251195 +0000 UTC m=+39.793664560" watchObservedRunningTime="2026-07-21 12:28:13.144075178 +0000 UTC m=+39.794488540" Jul 21 12:28:13.159959 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:13.159895 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t6fkr" podStartSLOduration=2.49514113 podStartE2EDuration="5.159879647s" totalImagesPullingTime="2.664738517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:13.159099318 +0000 UTC m=+39.809512679" watchObservedRunningTime="2026-07-21 12:28:13.159879647 +0000 UTC m=+39.810293008" Jul 21 12:28:13.592624 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:13.592586 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:13.592817 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:13.592713 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 12:28:13.592817 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:13.592787 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls podName:37341abd-787c-4e7c-a734-514a6a2ebaa0 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:15.592771918 +0000 UTC m=+42.243185260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-qrpqm" (UID: "37341abd-787c-4e7c-a734-514a6a2ebaa0") : secret "cluster-monitoring-operator-tls" not found Jul 21 12:28:13.795066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:13.794929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b42nb\" (UID: \"b1b704f0-bffb-48d7-bf16-ed9deea793f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" Jul 21 12:28:13.795248 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:13.795080 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 12:28:13.795248 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:13.795165 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls podName:b1b704f0-bffb-48d7-bf16-ed9deea793f0 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:15.795143571 +0000 UTC m=+42.445556926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b42nb" (UID: "b1b704f0-bffb-48d7-bf16-ed9deea793f0") : secret "samples-operator-tls" not found Jul 21 12:28:15.381718 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.381684 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-789d6579-qclzj"] Jul 21 12:28:15.395894 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.395854 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.396755 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.396728 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-789d6579-qclzj"] Jul 21 12:28:15.399540 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.398726 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 21 12:28:15.399540 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.398768 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 21 12:28:15.399540 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.398786 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bv9bb\"" Jul 21 12:28:15.399540 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.398731 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 21 12:28:15.405428 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.405392 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 21 12:28:15.510557 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.510522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.510720 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.510623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-certificates\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.510720 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.510642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-bound-sa-token\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.510720 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.510657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f7499fb-57aa-4c9f-a48c-45252cd451f1-installation-pull-secrets\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.510720 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.510679 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f7499fb-57aa-4c9f-a48c-45252cd451f1-trusted-ca\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.510720 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.510705 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f7499fb-57aa-4c9f-a48c-45252cd451f1-image-registry-private-configuration\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.510869 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.510754 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f7499fb-57aa-4c9f-a48c-45252cd451f1-ca-trust-extracted\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.510869 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.510778 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swscw\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-kube-api-access-swscw\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.611198 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.611152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:15.611371 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.611212 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-certificates\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.611371 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.611230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-bound-sa-token\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.611371 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.611250 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f7499fb-57aa-4c9f-a48c-45252cd451f1-installation-pull-secrets\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.611371 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.611306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f7499fb-57aa-4c9f-a48c-45252cd451f1-trusted-ca\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.611371 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:15.611324 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 12:28:15.611371 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.611355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f7499fb-57aa-4c9f-a48c-45252cd451f1-image-registry-private-configuration\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.611626 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.611382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f7499fb-57aa-4c9f-a48c-45252cd451f1-ca-trust-extracted\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.611626 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:15.611401 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls podName:37341abd-787c-4e7c-a734-514a6a2ebaa0 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:19.611379726 +0000 UTC m=+46.261793082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-qrpqm" (UID: "37341abd-787c-4e7c-a734-514a6a2ebaa0") : secret "cluster-monitoring-operator-tls" not found Jul 21 12:28:15.611626 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.611442 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swscw\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-kube-api-access-swscw\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.611626 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.611488 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.611779 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:15.611641 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 12:28:15.611779 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:15.611652 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-789d6579-qclzj: secret "image-registry-tls" not found Jul 21 12:28:15.611779 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.611683 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f7499fb-57aa-4c9f-a48c-45252cd451f1-ca-trust-extracted\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.611779 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:15.611689 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls podName:9f7499fb-57aa-4c9f-a48c-45252cd451f1 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:16.111676912 +0000 UTC m=+42.762090271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls") pod "image-registry-789d6579-qclzj" (UID: "9f7499fb-57aa-4c9f-a48c-45252cd451f1") : secret "image-registry-tls" not found Jul 21 12:28:15.611779 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.611762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-certificates\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.612554 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.612507 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f7499fb-57aa-4c9f-a48c-45252cd451f1-trusted-ca\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.615792 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.615771 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f7499fb-57aa-4c9f-a48c-45252cd451f1-installation-pull-secrets\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.615902 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.615825 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f7499fb-57aa-4c9f-a48c-45252cd451f1-image-registry-private-configuration\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.620045 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.620023 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swscw\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-kube-api-access-swscw\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.620141 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.620116 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-bound-sa-token\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:15.813774 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:15.813734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b42nb\" (UID: \"b1b704f0-bffb-48d7-bf16-ed9deea793f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" Jul 21 12:28:15.813967 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:15.813913 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 12:28:15.814021 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:15.814009 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls podName:b1b704f0-bffb-48d7-bf16-ed9deea793f0 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:19.813988735 +0000 UTC m=+46.464402089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b42nb" (UID: "b1b704f0-bffb-48d7-bf16-ed9deea793f0") : secret "samples-operator-tls" not found Jul 21 12:28:16.117307 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:16.117228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:16.117434 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:16.117400 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 12:28:16.117434 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:16.117422 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-789d6579-qclzj: secret "image-registry-tls" not found Jul 21 12:28:16.117516 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:16.117484 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls podName:9f7499fb-57aa-4c9f-a48c-45252cd451f1 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:17.117467054 +0000 UTC m=+43.767880397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls") pod "image-registry-789d6579-qclzj" (UID: "9f7499fb-57aa-4c9f-a48c-45252cd451f1") : secret "image-registry-tls" not found Jul 21 12:28:16.519679 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:16.519641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:16.520094 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:16.519813 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 12:28:16.520094 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:16.519893 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls podName:9de2e733-a5ab-429f-972e-b375324a5f82 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:24.519874917 +0000 UTC m=+51.170288263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls") pod "dns-default-c4jdq" (UID: "9de2e733-a5ab-429f-972e-b375324a5f82") : secret "dns-default-metrics-tls" not found Jul 21 12:28:17.124313 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:17.124272 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:17.124496 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:17.124438 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 12:28:17.124496 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:17.124464 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-789d6579-qclzj: secret "image-registry-tls" not found Jul 21 12:28:17.124590 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:17.124536 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls podName:9f7499fb-57aa-4c9f-a48c-45252cd451f1 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:19.124514677 +0000 UTC m=+45.774928024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls") pod "image-registry-789d6579-qclzj" (UID: "9f7499fb-57aa-4c9f-a48c-45252cd451f1") : secret "image-registry-tls" not found Jul 21 12:28:18.146019 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:18.145956 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n6plj" event={"ID":"4955ba99-c918-417c-8e62-85e8298bf82c","Type":"ContainerStarted","Data":"a0851ea985c64edced311aae6b0b92396a3b25d76be43b904a4a5322415fec5e"} Jul 21 12:28:18.148347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:18.148304 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-59v4g" event={"ID":"4d964404-b5ad-45f6-90c0-e70bb3986442","Type":"ContainerStarted","Data":"3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5"} Jul 21 12:28:18.150190 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:18.150137 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" event={"ID":"8947a59a-697b-41e4-9ecc-dfd60f9eab33","Type":"ContainerStarted","Data":"e81582eac78ddcc8945b9a7594ad2646f169beb132a767dd6d16c0bc03a45780"} Jul 21 12:28:18.154220 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:18.154192 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" event={"ID":"e4321016-2119-4390-8984-e74a3d4da956","Type":"ContainerStarted","Data":"e8232d8a26cd3942777cc2a191855f7dd1293955d12ae33708139010ef854d75"} Jul 21 12:28:18.154717 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:18.154641 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:18.156084 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:18.156056 2574 patch_prober.go:28] interesting pod/console-operator-565b7b569d-bc4k4 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.11:8443/readyz\": dial tcp 10.133.0.11:8443: connect: connection refused" start-of-body= Jul 21 12:28:18.156181 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:18.156110 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" podUID="e4321016-2119-4390-8984-e74a3d4da956" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.11:8443/readyz\": dial tcp 10.133.0.11:8443: connect: connection refused" Jul 21 12:28:18.157798 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:18.157362 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" event={"ID":"9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7","Type":"ContainerStarted","Data":"9efe10e84fea604ba07302f864c2abd2f77e18ff80f65547f71154b519b52bdb"} Jul 21 12:28:18.160990 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:18.160304 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n6plj" podStartSLOduration=2.144511736 podStartE2EDuration="7.160288778s" totalImagesPullingTime="5.015777042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:18.15998486 +0000 UTC m=+44.810398222" watchObservedRunningTime="2026-07-21 12:28:18.160288778 +0000 UTC m=+44.810702141" Jul 21 12:28:18.177241 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:18.177100 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" podStartSLOduration=2.082172059 podStartE2EDuration="7.177077798s" totalImagesPullingTime="5.094905739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:18.175694709 +0000 UTC m=+44.826108070" watchObservedRunningTime="2026-07-21 12:28:18.177077798 +0000 UTC m=+44.827491162" Jul 21 12:28:18.214046 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:18.213974 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" podStartSLOduration=2.153367296 podStartE2EDuration="7.213932824s" totalImagesPullingTime="5.060565528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:18.195249894 +0000 UTC m=+44.845663255" watchObservedRunningTime="2026-07-21 12:28:18.213932824 +0000 UTC m=+44.864346183" Jul 21 12:28:18.214916 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:18.214882 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-59v4g" podStartSLOduration=2.154782413 podStartE2EDuration="7.214873896s" totalImagesPullingTime="5.060091483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:18.213636418 +0000 UTC m=+44.864049779" watchObservedRunningTime="2026-07-21 12:28:18.214873896 +0000 UTC m=+44.865287273" Jul 21 12:28:18.239682 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:18.239634 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" podStartSLOduration=2.172935338 podStartE2EDuration="7.239616767s" totalImagesPullingTime="5.066681429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:18.238143502 +0000 UTC m=+44.888556859" watchObservedRunningTime="2026-07-21 12:28:18.239616767 +0000 UTC m=+44.890030106" Jul 21 12:28:19.146275 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.146238 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:19.146721 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:19.146420 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 12:28:19.146721 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:19.146439 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-789d6579-qclzj: secret "image-registry-tls" not found Jul 21 12:28:19.146721 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:19.146496 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls podName:9f7499fb-57aa-4c9f-a48c-45252cd451f1 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:23.146479327 +0000 UTC m=+49.796892666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls") pod "image-registry-789d6579-qclzj" (UID: "9f7499fb-57aa-4c9f-a48c-45252cd451f1") : secret "image-registry-tls" not found Jul 21 12:28:19.165318 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.165288 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/0.log" Jul 21 12:28:19.165491 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.165332 2574 generic.go:356] "Generic (PLEG): container finished" podID="e4321016-2119-4390-8984-e74a3d4da956" containerID="e8232d8a26cd3942777cc2a191855f7dd1293955d12ae33708139010ef854d75" exitCode=255 Jul 21 12:28:19.165491 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.165410 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" event={"ID":"e4321016-2119-4390-8984-e74a3d4da956","Type":"ContainerDied","Data":"e8232d8a26cd3942777cc2a191855f7dd1293955d12ae33708139010ef854d75"} Jul 21 12:28:19.165688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.165673 2574 scope.go:117] "RemoveContainer" containerID="e8232d8a26cd3942777cc2a191855f7dd1293955d12ae33708139010ef854d75" Jul 21 12:28:19.299995 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.299884 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9p2tq"] Jul 21 12:28:19.319510 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.319466 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9p2tq"] Jul 21 12:28:19.319738 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.319626 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:19.323314 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.322390 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 12:28:19.323314 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.322648 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 12:28:19.323314 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.322847 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c9cc7\"" Jul 21 12:28:19.450072 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.450032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e3e9900-040e-467b-91ce-18d7406625ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:19.450237 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.450085 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:19.450237 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.450146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65f4m\" (UniqueName: \"kubernetes.io/projected/0e3e9900-040e-467b-91ce-18d7406625ee-kube-api-access-65f4m\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:19.450237 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.450225 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e3e9900-040e-467b-91ce-18d7406625ee-data-volume\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:19.450349 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.450244 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e3e9900-040e-467b-91ce-18d7406625ee-crio-socket\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:19.551225 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.551135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65f4m\" (UniqueName: \"kubernetes.io/projected/0e3e9900-040e-467b-91ce-18d7406625ee-kube-api-access-65f4m\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:19.551373 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.551301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e3e9900-040e-467b-91ce-18d7406625ee-data-volume\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:19.551373 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.551335 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e3e9900-040e-467b-91ce-18d7406625ee-crio-socket\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:19.551441 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.551390 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e3e9900-040e-467b-91ce-18d7406625ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:19.551441 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.551408 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:19.551518 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.551491 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e3e9900-040e-467b-91ce-18d7406625ee-crio-socket\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:19.551518 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:19.551503 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 12:28:19.551577 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:19.551530 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 12:28:19.551577 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:19.551573 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e3e9900-040e-467b-91ce-18d7406625ee-kube-rbac-proxy-cm podName:0e3e9900-040e-467b-91ce-18d7406625ee nodeName:}" failed. No retries permitted until 2026-07-21 12:28:20.051550613 +0000 UTC m=+46.701963971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e3e9900-040e-467b-91ce-18d7406625ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9p2tq" (UID: "0e3e9900-040e-467b-91ce-18d7406625ee") : configmap "kube-rbac-proxy" not found Jul 21 12:28:19.551655 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:19.551590 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls podName:0e3e9900-040e-467b-91ce-18d7406625ee nodeName:}" failed. No retries permitted until 2026-07-21 12:28:20.051581 +0000 UTC m=+46.701994345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9p2tq" (UID: "0e3e9900-040e-467b-91ce-18d7406625ee") : secret "insights-runtime-extractor-tls" not found Jul 21 12:28:19.551691 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.551658 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e3e9900-040e-467b-91ce-18d7406625ee-data-volume\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:19.560391 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.560366 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65f4m\" (UniqueName: \"kubernetes.io/projected/0e3e9900-040e-467b-91ce-18d7406625ee-kube-api-access-65f4m\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:19.652495 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.652451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:19.652669 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:19.652603 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 12:28:19.652708 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:19.652679 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls podName:37341abd-787c-4e7c-a734-514a6a2ebaa0 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:27.652662628 +0000 UTC m=+54.303075970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-qrpqm" (UID: "37341abd-787c-4e7c-a734-514a6a2ebaa0") : secret "cluster-monitoring-operator-tls" not found Jul 21 12:28:19.854630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:19.854545 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b42nb\" (UID: \"b1b704f0-bffb-48d7-bf16-ed9deea793f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" Jul 21 12:28:19.854778 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:19.854695 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 12:28:19.854778 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:19.854760 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls podName:b1b704f0-bffb-48d7-bf16-ed9deea793f0 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:27.854745535 +0000 UTC m=+54.505158886 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b42nb" (UID: "b1b704f0-bffb-48d7-bf16-ed9deea793f0") : secret "samples-operator-tls" not found Jul 21 12:28:20.056038 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:20.056001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e3e9900-040e-467b-91ce-18d7406625ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:20.056038 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:20.056036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:20.056238 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:20.056130 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 12:28:20.056238 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:20.056173 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 12:28:20.056238 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:20.056201 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e3e9900-040e-467b-91ce-18d7406625ee-kube-rbac-proxy-cm podName:0e3e9900-040e-467b-91ce-18d7406625ee nodeName:}" failed. No retries permitted until 2026-07-21 12:28:21.056184154 +0000 UTC m=+47.706597493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e3e9900-040e-467b-91ce-18d7406625ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9p2tq" (UID: "0e3e9900-040e-467b-91ce-18d7406625ee") : configmap "kube-rbac-proxy" not found Jul 21 12:28:20.056238 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:20.056215 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls podName:0e3e9900-040e-467b-91ce-18d7406625ee nodeName:}" failed. No retries permitted until 2026-07-21 12:28:21.056209046 +0000 UTC m=+47.706622385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9p2tq" (UID: "0e3e9900-040e-467b-91ce-18d7406625ee") : secret "insights-runtime-extractor-tls" not found Jul 21 12:28:20.170765 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:20.170693 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:28:20.171180 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:20.171146 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/0.log" Jul 21 12:28:20.171222 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:20.171182 2574 generic.go:356] "Generic (PLEG): container finished" podID="e4321016-2119-4390-8984-e74a3d4da956" containerID="129862f7c504feb71ca77e98629d0c8cf34598f64dab256923b4b057cf312116" exitCode=255 Jul 21 12:28:20.171264 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:20.171229 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" event={"ID":"e4321016-2119-4390-8984-e74a3d4da956","Type":"ContainerDied","Data":"129862f7c504feb71ca77e98629d0c8cf34598f64dab256923b4b057cf312116"} Jul 21 12:28:20.171333 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:20.171298 2574 scope.go:117] "RemoveContainer" containerID="e8232d8a26cd3942777cc2a191855f7dd1293955d12ae33708139010ef854d75" Jul 21 12:28:20.171547 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:20.171531 2574 scope.go:117] "RemoveContainer" containerID="129862f7c504feb71ca77e98629d0c8cf34598f64dab256923b4b057cf312116" Jul 21 12:28:20.171712 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:20.171695 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-bc4k4_openshift-console-operator(e4321016-2119-4390-8984-e74a3d4da956)\"" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" podUID="e4321016-2119-4390-8984-e74a3d4da956" Jul 21 12:28:21.066057 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:21.066019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e3e9900-040e-467b-91ce-18d7406625ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:21.066057 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:21.066058 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:21.066269 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:21.066144 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 12:28:21.066269 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:21.066159 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 12:28:21.066269 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:21.066205 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e3e9900-040e-467b-91ce-18d7406625ee-kube-rbac-proxy-cm podName:0e3e9900-040e-467b-91ce-18d7406625ee nodeName:}" failed. No retries permitted until 2026-07-21 12:28:23.066190496 +0000 UTC m=+49.716603838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e3e9900-040e-467b-91ce-18d7406625ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9p2tq" (UID: "0e3e9900-040e-467b-91ce-18d7406625ee") : configmap "kube-rbac-proxy" not found Jul 21 12:28:21.066269 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:21.066220 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls podName:0e3e9900-040e-467b-91ce-18d7406625ee nodeName:}" failed. No retries permitted until 2026-07-21 12:28:23.066213556 +0000 UTC m=+49.716626895 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9p2tq" (UID: "0e3e9900-040e-467b-91ce-18d7406625ee") : secret "insights-runtime-extractor-tls" not found Jul 21 12:28:21.175861 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:21.175831 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:28:21.176249 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:21.176193 2574 scope.go:117] "RemoveContainer" containerID="129862f7c504feb71ca77e98629d0c8cf34598f64dab256923b4b057cf312116" Jul 21 12:28:21.176380 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:21.176360 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-bc4k4_openshift-console-operator(e4321016-2119-4390-8984-e74a3d4da956)\"" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" podUID="e4321016-2119-4390-8984-e74a3d4da956" Jul 21 12:28:21.246342 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:21.246310 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gqmm7_09deecb4-7e46-42cf-804d-634310995a12/dns-node-resolver/0.log" Jul 21 12:28:22.246985 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:22.246958 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lxclp_795fc54b-5129-4ef3-8fba-7aab06517e1f/node-ca/0.log" Jul 21 12:28:22.331593 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:22.331558 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:22.332049 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:22.332030 2574 scope.go:117] "RemoveContainer" containerID="129862f7c504feb71ca77e98629d0c8cf34598f64dab256923b4b057cf312116" Jul 21 12:28:22.332271 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:22.332248 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-bc4k4_openshift-console-operator(e4321016-2119-4390-8984-e74a3d4da956)\"" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" podUID="e4321016-2119-4390-8984-e74a3d4da956" Jul 21 12:28:23.049173 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:23.049143 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t6fkr_493eec89-2b07-4c39-a989-feea5c5a9a1b/serve-healthcheck-canary/0.log" Jul 21 12:28:23.086551 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:23.086511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e3e9900-040e-467b-91ce-18d7406625ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:23.086551 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:23.086551 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:23.086810 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:23.086683 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 12:28:23.086810 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:23.086788 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls podName:0e3e9900-040e-467b-91ce-18d7406625ee nodeName:}" failed. No retries permitted until 2026-07-21 12:28:27.086773202 +0000 UTC m=+53.737186542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9p2tq" (UID: "0e3e9900-040e-467b-91ce-18d7406625ee") : secret "insights-runtime-extractor-tls" not found Jul 21 12:28:23.087190 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:23.087163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e3e9900-040e-467b-91ce-18d7406625ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:23.186993 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:23.186957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:23.187164 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:23.187108 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 12:28:23.187164 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:23.187123 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-789d6579-qclzj: secret "image-registry-tls" not found Jul 21 12:28:23.187238 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:23.187187 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls podName:9f7499fb-57aa-4c9f-a48c-45252cd451f1 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:31.187170774 +0000 UTC m=+57.837584126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls") pod "image-registry-789d6579-qclzj" (UID: "9f7499fb-57aa-4c9f-a48c-45252cd451f1") : secret "image-registry-tls" not found Jul 21 12:28:24.598403 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:24.598362 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:24.598800 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:24.598519 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 12:28:24.598800 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:24.598597 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls podName:9de2e733-a5ab-429f-972e-b375324a5f82 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:40.598580252 +0000 UTC m=+67.248993591 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls") pod "dns-default-c4jdq" (UID: "9de2e733-a5ab-429f-972e-b375324a5f82") : secret "dns-default-metrics-tls" not found Jul 21 12:28:27.120033 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:27.120000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:27.120484 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:27.120136 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 12:28:27.120484 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:27.120204 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls podName:0e3e9900-040e-467b-91ce-18d7406625ee nodeName:}" failed. No retries permitted until 2026-07-21 12:28:35.120187927 +0000 UTC m=+61.770601266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9p2tq" (UID: "0e3e9900-040e-467b-91ce-18d7406625ee") : secret "insights-runtime-extractor-tls" not found Jul 21 12:28:27.724687 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:27.724639 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:27.724867 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:27.724800 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 12:28:27.724908 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:27.724868 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls podName:37341abd-787c-4e7c-a734-514a6a2ebaa0 nodeName:}" failed. No retries permitted until 2026-07-21 12:28:43.724851841 +0000 UTC m=+70.375265180 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-qrpqm" (UID: "37341abd-787c-4e7c-a734-514a6a2ebaa0") : secret "cluster-monitoring-operator-tls" not found Jul 21 12:28:27.927271 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:27.927229 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b42nb\" (UID: \"b1b704f0-bffb-48d7-bf16-ed9deea793f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" Jul 21 12:28:27.929590 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:27.929560 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1b704f0-bffb-48d7-bf16-ed9deea793f0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b42nb\" (UID: \"b1b704f0-bffb-48d7-bf16-ed9deea793f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" Jul 21 12:28:27.941571 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:27.941542 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" Jul 21 12:28:28.067546 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:28.067504 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb"] Jul 21 12:28:28.154823 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:28.154788 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:28.155210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:28.155179 2574 scope.go:117] "RemoveContainer" containerID="129862f7c504feb71ca77e98629d0c8cf34598f64dab256923b4b057cf312116" Jul 21 12:28:28.155365 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:28:28.155345 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-bc4k4_openshift-console-operator(e4321016-2119-4390-8984-e74a3d4da956)\"" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" podUID="e4321016-2119-4390-8984-e74a3d4da956" Jul 21 12:28:28.200752 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:28.200714 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" event={"ID":"b1b704f0-bffb-48d7-bf16-ed9deea793f0","Type":"ContainerStarted","Data":"444b13e2dd7a7f129a6c01bc4fe26ddf97bf5c4cf051a4f0dda729fa04395562"} Jul 21 12:28:30.211067 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:30.211022 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" event={"ID":"b1b704f0-bffb-48d7-bf16-ed9deea793f0","Type":"ContainerStarted","Data":"a4f7f985cdbebf58ed605e0f48e86c338581a7c7682fa74e952f9c309475a969"} Jul 21 12:28:30.211501 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:30.211088 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" event={"ID":"b1b704f0-bffb-48d7-bf16-ed9deea793f0","Type":"ContainerStarted","Data":"eb2993d3c3cf682507496336a1a589af567447123e7916bc06d8a7768fe1b85c"} Jul 21 12:28:30.232896 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:30.232802 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b42nb" podStartSLOduration=17.357405232 podStartE2EDuration="19.232785363s" totalImagesPullingTime="1.875380131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:30.232130941 +0000 UTC m=+56.882544303" watchObservedRunningTime="2026-07-21 12:28:30.232785363 +0000 UTC m=+56.883198724" Jul 21 12:28:31.258049 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:31.257997 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:31.260409 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:31.260383 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls\") pod \"image-registry-789d6579-qclzj\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:31.309006 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:31.308963 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:31.439703 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:31.439675 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-789d6579-qclzj"] Jul 21 12:28:31.442685 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:28:31.442642 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7499fb_57aa_4c9f_a48c_45252cd451f1.slice/crio-83123fffbc76231d34760d3721a5d39ada88ce7f517a86c96b0d3b6a9c6e36b8 WatchSource:0}: Error finding container 83123fffbc76231d34760d3721a5d39ada88ce7f517a86c96b0d3b6a9c6e36b8: Status 404 returned error can't find the container with id 83123fffbc76231d34760d3721a5d39ada88ce7f517a86c96b0d3b6a9c6e36b8 Jul 21 12:28:32.218644 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:32.218611 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-789d6579-qclzj" event={"ID":"9f7499fb-57aa-4c9f-a48c-45252cd451f1","Type":"ContainerStarted","Data":"e61eb4eb2c5c6029da105ca71980d1e21735742b24bba8536fba8a2d34b3ddda"} Jul 21 12:28:32.218836 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:32.218686 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-789d6579-qclzj" event={"ID":"9f7499fb-57aa-4c9f-a48c-45252cd451f1","Type":"ContainerStarted","Data":"83123fffbc76231d34760d3721a5d39ada88ce7f517a86c96b0d3b6a9c6e36b8"} Jul 21 12:28:32.218836 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:32.218784 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:32.245345 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:32.245287 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-789d6579-qclzj" podStartSLOduration=17.245266185 podStartE2EDuration="17.245266185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:32.244432894 +0000 UTC m=+58.894846258" watchObservedRunningTime="2026-07-21 12:28:32.245266185 +0000 UTC m=+58.895679545" Jul 21 12:28:34.094884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:34.094859 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhsp9" Jul 21 12:28:35.191730 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:35.191699 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:35.193851 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:35.193828 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e3e9900-040e-467b-91ce-18d7406625ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p2tq\" (UID: \"0e3e9900-040e-467b-91ce-18d7406625ee\") " pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:35.234853 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:35.234822 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c9cc7\"" Jul 21 12:28:35.243074 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:35.243050 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9p2tq" Jul 21 12:28:35.372353 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:35.372321 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9p2tq"] Jul 21 12:28:35.375807 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:28:35.375773 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3e9900_040e_467b_91ce_18d7406625ee.slice/crio-a5521ff8ee617ed0db370661af88413be06dc940e6fcf24e27453f2f8c493513 WatchSource:0}: Error finding container a5521ff8ee617ed0db370661af88413be06dc940e6fcf24e27453f2f8c493513: Status 404 returned error can't find the container with id a5521ff8ee617ed0db370661af88413be06dc940e6fcf24e27453f2f8c493513 Jul 21 12:28:36.232101 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:36.232066 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9p2tq" event={"ID":"0e3e9900-040e-467b-91ce-18d7406625ee","Type":"ContainerStarted","Data":"a5521ff8ee617ed0db370661af88413be06dc940e6fcf24e27453f2f8c493513"} Jul 21 12:28:38.918734 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:38.918703 2574 scope.go:117] "RemoveContainer" containerID="129862f7c504feb71ca77e98629d0c8cf34598f64dab256923b4b057cf312116" Jul 21 12:28:39.630957 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:39.630901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs\") pod \"network-metrics-daemon-vzml4\" (UID: \"9559d0ad-feb3-4e5b-95e8-97173ecf07eb\") " pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:28:39.631156 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:39.631024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:28:39.634101 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:39.634077 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 12:28:39.634246 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:39.634127 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 12:28:39.644016 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:39.643969 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0be2da5-e2d6-4bae-bdcc-c65eb877fa03-original-pull-secret\") pod \"global-pull-secret-syncer-rz4f4\" (UID: \"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03\") " pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:28:39.644016 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:39.643993 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9559d0ad-feb3-4e5b-95e8-97173ecf07eb-metrics-certs\") pod \"network-metrics-daemon-vzml4\" (UID: \"9559d0ad-feb3-4e5b-95e8-97173ecf07eb\") " pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:28:39.830132 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:39.830095 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz4f4" Jul 21 12:28:39.832624 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:39.832591 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrrb\" (UniqueName: \"kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb\") pod \"network-check-target-xjxkh\" (UID: \"c4f3ee5b-f5e6-48eb-8820-2a23cc003565\") " pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:28:39.836110 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:39.836088 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 12:28:39.838592 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:39.838564 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-288sp\"" Jul 21 12:28:39.845386 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:39.845361 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 12:28:39.846326 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:39.846305 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzml4" Jul 21 12:28:39.857250 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:39.857224 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsrrb\" (UniqueName: \"kubernetes.io/projected/c4f3ee5b-f5e6-48eb-8820-2a23cc003565-kube-api-access-fsrrb\") pod \"network-check-target-xjxkh\" (UID: \"c4f3ee5b-f5e6-48eb-8820-2a23cc003565\") " pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:28:40.145787 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:40.145756 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mz2qd\"" Jul 21 12:28:40.153335 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:40.153305 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:28:40.638254 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:40.638219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:40.640881 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:40.640852 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de2e733-a5ab-429f-972e-b375324a5f82-metrics-tls\") pod \"dns-default-c4jdq\" (UID: \"9de2e733-a5ab-429f-972e-b375324a5f82\") " pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:40.841510 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:40.841476 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r5psm\"" Jul 21 12:28:40.848893 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:40.848867 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:41.038578 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:41.038547 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rz4f4"] Jul 21 12:28:41.197461 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:28:41.197384 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0be2da5_e2d6_4bae_bdcc_c65eb877fa03.slice/crio-5396191f92ed53728235c245d090e0d8d74f92b0d479adfe84fbc16be5cf46b8 WatchSource:0}: Error finding container 5396191f92ed53728235c245d090e0d8d74f92b0d479adfe84fbc16be5cf46b8: Status 404 returned error can't find the container with id 5396191f92ed53728235c245d090e0d8d74f92b0d479adfe84fbc16be5cf46b8 Jul 21 12:28:41.249956 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:41.249911 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rz4f4" event={"ID":"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03","Type":"ContainerStarted","Data":"5396191f92ed53728235c245d090e0d8d74f92b0d479adfe84fbc16be5cf46b8"} Jul 21 12:28:41.251768 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:41.251749 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:28:41.251882 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:41.251799 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" event={"ID":"e4321016-2119-4390-8984-e74a3d4da956","Type":"ContainerStarted","Data":"cc386d0bf6a185dedf68ad983b6d70cb5e0147b07042f105f2560cc0514ec4c0"} Jul 21 12:28:41.259194 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:41.259171 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vzml4"] Jul 21 12:28:41.262162 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:41.262123 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xjxkh"] Jul 21 12:28:41.262329 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:28:41.262307 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9559d0ad_feb3_4e5b_95e8_97173ecf07eb.slice/crio-1bc6a404c1975334dba5753557da674dbb6b4d04145a0aa9f8341b946395e67b WatchSource:0}: Error finding container 1bc6a404c1975334dba5753557da674dbb6b4d04145a0aa9f8341b946395e67b: Status 404 returned error can't find the container with id 1bc6a404c1975334dba5753557da674dbb6b4d04145a0aa9f8341b946395e67b Jul 21 12:28:41.266609 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:41.266586 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c4jdq"] Jul 21 12:28:41.270051 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:28:41.270026 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de2e733_a5ab_429f_972e_b375324a5f82.slice/crio-f868ec55e5956f64052b13536b6baef5b44f5ff237972eb3f28246daac4484e8 WatchSource:0}: Error finding container f868ec55e5956f64052b13536b6baef5b44f5ff237972eb3f28246daac4484e8: Status 404 returned error can't find the container with id f868ec55e5956f64052b13536b6baef5b44f5ff237972eb3f28246daac4484e8 Jul 21 12:28:42.259740 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:42.259637 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9p2tq" event={"ID":"0e3e9900-040e-467b-91ce-18d7406625ee","Type":"ContainerStarted","Data":"edb6b4f98cc6d65a1f4c812aef041d1c5353d34bed3a44133e8048c0521ce5ae"} Jul 21 12:28:42.261662 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:42.261633 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xjxkh" event={"ID":"c4f3ee5b-f5e6-48eb-8820-2a23cc003565","Type":"ContainerStarted","Data":"1e074fbba77ccb8059645bf90a6ba23048cb53c6117a9370f9915c75c45ef348"} Jul 21 12:28:42.263318 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:42.263286 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4jdq" event={"ID":"9de2e733-a5ab-429f-972e-b375324a5f82","Type":"ContainerStarted","Data":"f868ec55e5956f64052b13536b6baef5b44f5ff237972eb3f28246daac4484e8"} Jul 21 12:28:42.265098 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:42.265071 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vzml4" event={"ID":"9559d0ad-feb3-4e5b-95e8-97173ecf07eb","Type":"ContainerStarted","Data":"1bc6a404c1975334dba5753557da674dbb6b4d04145a0aa9f8341b946395e67b"} Jul 21 12:28:43.771995 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:43.771932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:43.775459 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:43.775421 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/37341abd-787c-4e7c-a734-514a6a2ebaa0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qrpqm\" (UID: \"37341abd-787c-4e7c-a734-514a6a2ebaa0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:44.014747 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.014715 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hdckv\"" Jul 21 12:28:44.016868 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.016838 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" Jul 21 12:28:44.167062 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.166975 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62"] Jul 21 12:28:44.169006 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.168982 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" Jul 21 12:28:44.181041 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.181003 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 21 12:28:44.182253 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.182144 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 21 12:28:44.182253 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.182154 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 21 12:28:44.185223 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.185199 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 21 12:28:44.277419 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.277382 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df143a2a-c496-4bfa-8827-c076ed24b1ff-tmp\") pod \"klusterlet-addon-workmgr-7f5ccbff8d-vcr62\" (UID: \"df143a2a-c496-4bfa-8827-c076ed24b1ff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" Jul 21 12:28:44.277619 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.277464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrh59\" (UniqueName: \"kubernetes.io/projected/df143a2a-c496-4bfa-8827-c076ed24b1ff-kube-api-access-wrh59\") pod \"klusterlet-addon-workmgr-7f5ccbff8d-vcr62\" (UID: \"df143a2a-c496-4bfa-8827-c076ed24b1ff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" Jul 21 12:28:44.277619 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.277518 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/df143a2a-c496-4bfa-8827-c076ed24b1ff-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f5ccbff8d-vcr62\" (UID: \"df143a2a-c496-4bfa-8827-c076ed24b1ff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" Jul 21 12:28:44.320479 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.320443 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-789d6579-qclzj"] Jul 21 12:28:44.323062 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.323032 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62"] Jul 21 12:28:44.377912 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.377870 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/df143a2a-c496-4bfa-8827-c076ed24b1ff-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f5ccbff8d-vcr62\" (UID: \"df143a2a-c496-4bfa-8827-c076ed24b1ff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" Jul 21 12:28:44.378560 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.377926 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df143a2a-c496-4bfa-8827-c076ed24b1ff-tmp\") pod \"klusterlet-addon-workmgr-7f5ccbff8d-vcr62\" (UID: \"df143a2a-c496-4bfa-8827-c076ed24b1ff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" Jul 21 12:28:44.378560 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.378003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrh59\" (UniqueName: \"kubernetes.io/projected/df143a2a-c496-4bfa-8827-c076ed24b1ff-kube-api-access-wrh59\") pod \"klusterlet-addon-workmgr-7f5ccbff8d-vcr62\" (UID: \"df143a2a-c496-4bfa-8827-c076ed24b1ff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" Jul 21 12:28:44.378560 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.378355 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df143a2a-c496-4bfa-8827-c076ed24b1ff-tmp\") pod \"klusterlet-addon-workmgr-7f5ccbff8d-vcr62\" (UID: \"df143a2a-c496-4bfa-8827-c076ed24b1ff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" Jul 21 12:28:44.379475 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.379318 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d946d445-xgmq9"] Jul 21 12:28:44.381173 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.381152 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/df143a2a-c496-4bfa-8827-c076ed24b1ff-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f5ccbff8d-vcr62\" (UID: \"df143a2a-c496-4bfa-8827-c076ed24b1ff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" Jul 21 12:28:44.382892 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.382659 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.400291 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.400166 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d946d445-xgmq9"] Jul 21 12:28:44.404562 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.404527 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrh59\" (UniqueName: \"kubernetes.io/projected/df143a2a-c496-4bfa-8827-c076ed24b1ff-kube-api-access-wrh59\") pod \"klusterlet-addon-workmgr-7f5ccbff8d-vcr62\" (UID: \"df143a2a-c496-4bfa-8827-c076ed24b1ff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" Jul 21 12:28:44.478789 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.478702 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01f4155a-f514-4452-b5c7-23bc7db245f0-image-registry-private-configuration\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.478789 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.478746 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01f4155a-f514-4452-b5c7-23bc7db245f0-registry-certificates\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.478789 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.478773 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01f4155a-f514-4452-b5c7-23bc7db245f0-installation-pull-secrets\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.479068 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.478829 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f4155a-f514-4452-b5c7-23bc7db245f0-registry-tls\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.479068 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.478866 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01f4155a-f514-4452-b5c7-23bc7db245f0-bound-sa-token\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.479068 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.478893 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01f4155a-f514-4452-b5c7-23bc7db245f0-trusted-ca\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.479068 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.478929 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01f4155a-f514-4452-b5c7-23bc7db245f0-ca-trust-extracted\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.479068 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.478968 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldgd8\" (UniqueName: \"kubernetes.io/projected/01f4155a-f514-4452-b5c7-23bc7db245f0-kube-api-access-ldgd8\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.480121 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.480085 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" Jul 21 12:28:44.579874 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.579827 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01f4155a-f514-4452-b5c7-23bc7db245f0-bound-sa-token\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.580064 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.579881 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01f4155a-f514-4452-b5c7-23bc7db245f0-trusted-ca\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.580064 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.579929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01f4155a-f514-4452-b5c7-23bc7db245f0-ca-trust-extracted\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.580064 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.579973 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldgd8\" (UniqueName: \"kubernetes.io/projected/01f4155a-f514-4452-b5c7-23bc7db245f0-kube-api-access-ldgd8\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.580064 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.580028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01f4155a-f514-4452-b5c7-23bc7db245f0-image-registry-private-configuration\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.580064 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.580058 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01f4155a-f514-4452-b5c7-23bc7db245f0-registry-certificates\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.580339 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.580088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01f4155a-f514-4452-b5c7-23bc7db245f0-installation-pull-secrets\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.580339 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.580149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f4155a-f514-4452-b5c7-23bc7db245f0-registry-tls\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.580440 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.580421 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01f4155a-f514-4452-b5c7-23bc7db245f0-ca-trust-extracted\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.581351 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.581298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01f4155a-f514-4452-b5c7-23bc7db245f0-registry-certificates\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.581706 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.581687 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01f4155a-f514-4452-b5c7-23bc7db245f0-trusted-ca\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.583789 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.583734 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01f4155a-f514-4452-b5c7-23bc7db245f0-installation-pull-secrets\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.584036 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.584014 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01f4155a-f514-4452-b5c7-23bc7db245f0-image-registry-private-configuration\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.584203 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.584178 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f4155a-f514-4452-b5c7-23bc7db245f0-registry-tls\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.591569 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.591541 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01f4155a-f514-4452-b5c7-23bc7db245f0-bound-sa-token\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.597306 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.597270 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldgd8\" (UniqueName: \"kubernetes.io/projected/01f4155a-f514-4452-b5c7-23bc7db245f0-kube-api-access-ldgd8\") pod \"image-registry-7d946d445-xgmq9\" (UID: \"01f4155a-f514-4452-b5c7-23bc7db245f0\") " pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:44.700341 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:44.700297 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:48.584410 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:48.584366 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm"] Jul 21 12:28:48.756435 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:48.756409 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62"] Jul 21 12:28:48.760169 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:28:48.759908 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf143a2a_c496_4bfa_8827_c076ed24b1ff.slice/crio-1603bd5889fd6da225bdee5ce1cb2cb7eef47991c43069369d883dcce6756187 WatchSource:0}: Error finding container 1603bd5889fd6da225bdee5ce1cb2cb7eef47991c43069369d883dcce6756187: Status 404 returned error can't find the container with id 1603bd5889fd6da225bdee5ce1cb2cb7eef47991c43069369d883dcce6756187 Jul 21 12:28:48.786516 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:48.786481 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d946d445-xgmq9"] Jul 21 12:28:49.309150 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.309092 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9p2tq" event={"ID":"0e3e9900-040e-467b-91ce-18d7406625ee","Type":"ContainerStarted","Data":"d68a3b0eb1ef93490a7325c54a33746813a085c49cdb3b0881613651dcb50526"} Jul 21 12:28:49.312233 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.312137 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xjxkh" event={"ID":"c4f3ee5b-f5e6-48eb-8820-2a23cc003565","Type":"ContainerStarted","Data":"9f2cb3af97e60b6bf21a5fa6c44bdc1c6227a50e586cf46d9fd7adc10da224ff"} Jul 21 12:28:49.312376 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.312301 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:28:49.313922 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.313895 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d946d445-xgmq9" event={"ID":"01f4155a-f514-4452-b5c7-23bc7db245f0","Type":"ContainerStarted","Data":"da0ff151e7d0d7aae82509025e2f80e08fef643d534ddd14e81bb6af1dd42e03"} Jul 21 12:28:49.314059 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.313962 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d946d445-xgmq9" event={"ID":"01f4155a-f514-4452-b5c7-23bc7db245f0","Type":"ContainerStarted","Data":"79d440efb0c57a57d1532826ca9313570378920ad8d1a17a635bb89b75940fd0"} Jul 21 12:28:49.314059 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.313992 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:28:49.315552 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.315523 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rz4f4" event={"ID":"b0be2da5-e2d6-4bae-bdcc-c65eb877fa03","Type":"ContainerStarted","Data":"303eabca15e84743451bc301d1495497315018bea99022feb242de0a38a64cec"} Jul 21 12:28:49.317564 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.317540 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4jdq" event={"ID":"9de2e733-a5ab-429f-972e-b375324a5f82","Type":"ContainerStarted","Data":"c43b20ca8f5ffc9744b3a64c670d6eb2f63439640cc144e2f0945c4512f699c2"} Jul 21 12:28:49.317662 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.317582 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4jdq" event={"ID":"9de2e733-a5ab-429f-972e-b375324a5f82","Type":"ContainerStarted","Data":"671d4f1242ab6334aaa7b735b0b16e23733dbe39620d9fd2282f767f083d363c"} Jul 21 12:28:49.317662 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.317604 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c4jdq" Jul 21 12:28:49.320812 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.320343 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vzml4" event={"ID":"9559d0ad-feb3-4e5b-95e8-97173ecf07eb","Type":"ContainerStarted","Data":"7d9abd5f471737a75ede02c0a2ef6b628ec6f66d338e56d688fd4623dc73956b"} Jul 21 12:28:49.320812 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.320387 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vzml4" event={"ID":"9559d0ad-feb3-4e5b-95e8-97173ecf07eb","Type":"ContainerStarted","Data":"2ebd56f69224b6368f60b4ff23424d2a5b6b413a63becdbc0c600d2a36dc7d51"} Jul 21 12:28:49.322702 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.322678 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" event={"ID":"df143a2a-c496-4bfa-8827-c076ed24b1ff","Type":"ContainerStarted","Data":"1603bd5889fd6da225bdee5ce1cb2cb7eef47991c43069369d883dcce6756187"} Jul 21 12:28:49.323997 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.323954 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" event={"ID":"37341abd-787c-4e7c-a734-514a6a2ebaa0","Type":"ContainerStarted","Data":"3fb0470b721092020373164ce10c0cd5512a6e996da880d4519d48e2985e25dc"} Jul 21 12:28:49.330026 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.329531 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xjxkh" podStartSLOduration=68.044325558 podStartE2EDuration="1m15.329513673s" totalImagesPullingTime="7.285188115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:49.328124343 +0000 UTC m=+75.978537722" watchObservedRunningTime="2026-07-21 12:28:49.329513673 +0000 UTC m=+75.979927036" Jul 21 12:28:49.346093 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.346035 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c4jdq" podStartSLOduration=34.170840053 podStartE2EDuration="41.346018037s" totalImagesPullingTime="7.175177984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:49.345643266 +0000 UTC m=+75.996056633" watchObservedRunningTime="2026-07-21 12:28:49.346018037 +0000 UTC m=+75.996431397" Jul 21 12:28:49.363496 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.363377 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rz4f4" podStartSLOduration=68.03366358 podStartE2EDuration="1m15.36335426s" totalImagesPullingTime="7.32969068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:49.361913195 +0000 UTC m=+76.012326556" watchObservedRunningTime="2026-07-21 12:28:49.36335426 +0000 UTC m=+76.013767622" Jul 21 12:28:49.380486 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.380427 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vzml4" podStartSLOduration=68.20829285 podStartE2EDuration="1m15.380406104s" totalImagesPullingTime="7.172113254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:49.379114146 +0000 UTC m=+76.029527509" watchObservedRunningTime="2026-07-21 12:28:49.380406104 +0000 UTC m=+76.030819478" Jul 21 12:28:49.432466 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:49.431677 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d946d445-xgmq9" podStartSLOduration=5.431658541 podStartE2EDuration="5.431658541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:49.431238306 +0000 UTC m=+76.081651667" watchObservedRunningTime="2026-07-21 12:28:49.431658541 +0000 UTC m=+76.082071903" Jul 21 12:28:51.265158 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:51.265117 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:51.269313 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:51.269284 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-bc4k4" Jul 21 12:28:54.326647 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:54.326620 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:28:55.359298 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:55.359171 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" event={"ID":"df143a2a-c496-4bfa-8827-c076ed24b1ff","Type":"ContainerStarted","Data":"71fbfe1ef7be647da06ed34b68130992a4636f67236c7e87c12e10418256f119"} Jul 21 12:28:55.359737 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:55.359366 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" Jul 21 12:28:55.361220 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:55.361181 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" event={"ID":"37341abd-787c-4e7c-a734-514a6a2ebaa0","Type":"ContainerStarted","Data":"8b45a470a34d469b9ac10f9c5fd088cffc106220accdb5a72230c8bab1e49dc0"} Jul 21 12:28:55.361442 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:55.361411 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" Jul 21 12:28:55.363409 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:55.363376 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9p2tq" event={"ID":"0e3e9900-040e-467b-91ce-18d7406625ee","Type":"ContainerStarted","Data":"bc1ab9083308103910fbfeb0bb644b44b091003a592dedaf4225fe88959e6237"} Jul 21 12:28:55.376874 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:55.376831 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5ccbff8d-vcr62" podStartSLOduration=5.319139446 podStartE2EDuration="11.376815179s" totalImagesPullingTime="6.057675733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:55.375847802 +0000 UTC m=+82.026261177" watchObservedRunningTime="2026-07-21 12:28:55.376815179 +0000 UTC m=+82.027228541" Jul 21 12:28:55.394245 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:55.394188 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9p2tq" podStartSLOduration=17.187741166 podStartE2EDuration="36.394174434s" totalImagesPullingTime="19.206433268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:28:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:55.392700522 +0000 UTC m=+82.043113886" watchObservedRunningTime="2026-07-21 12:28:55.394174434 +0000 UTC m=+82.044587795" Jul 21 12:28:55.423603 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:55.423551 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qrpqm" podStartSLOduration=38.218148067 podStartE2EDuration="44.423534827s" totalImagesPullingTime="6.20538676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:28:55.423055729 +0000 UTC m=+82.073469091" watchObservedRunningTime="2026-07-21 12:28:55.423534827 +0000 UTC m=+82.073948187" Jul 21 12:28:59.333031 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:28:59.332995 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c4jdq" Jul 21 12:29:09.349444 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.349365 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-789d6579-qclzj" podUID="9f7499fb-57aa-4c9f-a48c-45252cd451f1" containerName="registry" containerID="cri-o://e61eb4eb2c5c6029da105ca71980d1e21735742b24bba8536fba8a2d34b3ddda" gracePeriod=30 Jul 21 12:29:09.593261 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.593237 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:29:09.692881 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.692795 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f7499fb-57aa-4c9f-a48c-45252cd451f1-trusted-ca\") pod \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " Jul 21 12:29:09.692881 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.692848 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f7499fb-57aa-4c9f-a48c-45252cd451f1-image-registry-private-configuration\") pod \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " Jul 21 12:29:09.692881 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.692873 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls\") pod \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " Jul 21 12:29:09.693190 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.692891 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-bound-sa-token\") pod \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " Jul 21 12:29:09.693190 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.692930 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-certificates\") pod \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " Jul 21 12:29:09.693190 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.692982 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f7499fb-57aa-4c9f-a48c-45252cd451f1-installation-pull-secrets\") pod \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " Jul 21 12:29:09.693190 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.693017 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swscw\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-kube-api-access-swscw\") pod \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " Jul 21 12:29:09.693190 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.693047 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f7499fb-57aa-4c9f-a48c-45252cd451f1-ca-trust-extracted\") pod \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\" (UID: \"9f7499fb-57aa-4c9f-a48c-45252cd451f1\") " Jul 21 12:29:09.693425 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.693302 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7499fb-57aa-4c9f-a48c-45252cd451f1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9f7499fb-57aa-4c9f-a48c-45252cd451f1" (UID: "9f7499fb-57aa-4c9f-a48c-45252cd451f1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:29:09.693525 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.693493 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9f7499fb-57aa-4c9f-a48c-45252cd451f1" (UID: "9f7499fb-57aa-4c9f-a48c-45252cd451f1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:29:09.695599 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.695566 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f7499fb-57aa-4c9f-a48c-45252cd451f1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9f7499fb-57aa-4c9f-a48c-45252cd451f1" (UID: "9f7499fb-57aa-4c9f-a48c-45252cd451f1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:29:09.695723 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.695644 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9f7499fb-57aa-4c9f-a48c-45252cd451f1" (UID: "9f7499fb-57aa-4c9f-a48c-45252cd451f1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:29:09.695723 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.695674 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f7499fb-57aa-4c9f-a48c-45252cd451f1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9f7499fb-57aa-4c9f-a48c-45252cd451f1" (UID: "9f7499fb-57aa-4c9f-a48c-45252cd451f1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:29:09.695812 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.695758 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9f7499fb-57aa-4c9f-a48c-45252cd451f1" (UID: "9f7499fb-57aa-4c9f-a48c-45252cd451f1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:29:09.695856 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.695817 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-kube-api-access-swscw" (OuterVolumeSpecName: "kube-api-access-swscw") pod "9f7499fb-57aa-4c9f-a48c-45252cd451f1" (UID: "9f7499fb-57aa-4c9f-a48c-45252cd451f1"). InnerVolumeSpecName "kube-api-access-swscw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:29:09.701653 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.701629 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7499fb-57aa-4c9f-a48c-45252cd451f1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9f7499fb-57aa-4c9f-a48c-45252cd451f1" (UID: "9f7499fb-57aa-4c9f-a48c-45252cd451f1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:29:09.794207 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.794173 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f7499fb-57aa-4c9f-a48c-45252cd451f1-image-registry-private-configuration\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:29:09.794207 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.794202 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-tls\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:29:09.794207 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.794212 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-bound-sa-token\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:29:09.794432 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.794220 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f7499fb-57aa-4c9f-a48c-45252cd451f1-registry-certificates\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:29:09.794432 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.794229 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f7499fb-57aa-4c9f-a48c-45252cd451f1-installation-pull-secrets\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:29:09.794432 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.794237 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swscw\" (UniqueName: \"kubernetes.io/projected/9f7499fb-57aa-4c9f-a48c-45252cd451f1-kube-api-access-swscw\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:29:09.794432 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.794245 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f7499fb-57aa-4c9f-a48c-45252cd451f1-ca-trust-extracted\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:29:09.794432 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:09.794253 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f7499fb-57aa-4c9f-a48c-45252cd451f1-trusted-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:29:10.335531 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:10.335502 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d946d445-xgmq9" Jul 21 12:29:10.417892 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:10.417859 2574 generic.go:356] "Generic (PLEG): container finished" podID="9f7499fb-57aa-4c9f-a48c-45252cd451f1" containerID="e61eb4eb2c5c6029da105ca71980d1e21735742b24bba8536fba8a2d34b3ddda" exitCode=0 Jul 21 12:29:10.418332 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:10.417894 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-789d6579-qclzj" event={"ID":"9f7499fb-57aa-4c9f-a48c-45252cd451f1","Type":"ContainerDied","Data":"e61eb4eb2c5c6029da105ca71980d1e21735742b24bba8536fba8a2d34b3ddda"} Jul 21 12:29:10.418332 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:10.417915 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-789d6579-qclzj" Jul 21 12:29:10.418332 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:10.417961 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-789d6579-qclzj" event={"ID":"9f7499fb-57aa-4c9f-a48c-45252cd451f1","Type":"ContainerDied","Data":"83123fffbc76231d34760d3721a5d39ada88ce7f517a86c96b0d3b6a9c6e36b8"} Jul 21 12:29:10.418332 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:10.417980 2574 scope.go:117] "RemoveContainer" containerID="e61eb4eb2c5c6029da105ca71980d1e21735742b24bba8536fba8a2d34b3ddda" Jul 21 12:29:10.426405 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:10.426386 2574 scope.go:117] "RemoveContainer" containerID="e61eb4eb2c5c6029da105ca71980d1e21735742b24bba8536fba8a2d34b3ddda" Jul 21 12:29:10.426729 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:29:10.426699 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61eb4eb2c5c6029da105ca71980d1e21735742b24bba8536fba8a2d34b3ddda\": container with ID starting with e61eb4eb2c5c6029da105ca71980d1e21735742b24bba8536fba8a2d34b3ddda not found: ID does not exist" containerID="e61eb4eb2c5c6029da105ca71980d1e21735742b24bba8536fba8a2d34b3ddda" Jul 21 12:29:10.426804 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:10.426736 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61eb4eb2c5c6029da105ca71980d1e21735742b24bba8536fba8a2d34b3ddda"} err="failed to get container status \"e61eb4eb2c5c6029da105ca71980d1e21735742b24bba8536fba8a2d34b3ddda\": rpc error: code = NotFound desc = could not find container \"e61eb4eb2c5c6029da105ca71980d1e21735742b24bba8536fba8a2d34b3ddda\": container with ID starting with e61eb4eb2c5c6029da105ca71980d1e21735742b24bba8536fba8a2d34b3ddda not found: ID does not exist" Jul 21 12:29:10.441724 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:10.441697 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-789d6579-qclzj"] Jul 21 12:29:10.448917 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:10.448894 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-789d6579-qclzj"] Jul 21 12:29:11.779182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.779144 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qq577"] Jul 21 12:29:11.779551 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.779399 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7499fb-57aa-4c9f-a48c-45252cd451f1" containerName="registry" Jul 21 12:29:11.779551 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.779409 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7499fb-57aa-4c9f-a48c-45252cd451f1" containerName="registry" Jul 21 12:29:11.779551 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.779448 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7499fb-57aa-4c9f-a48c-45252cd451f1" containerName="registry" Jul 21 12:29:11.784594 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.784573 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:11.787200 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.787178 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 12:29:11.787334 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.787265 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 12:29:11.787334 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.787317 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sjw6b\"" Jul 21 12:29:11.788501 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.788482 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 12:29:11.788607 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.788515 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 12:29:11.910708 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.910679 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-tls\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:11.910708 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.910713 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdxqr\" (UniqueName: \"kubernetes.io/projected/0a03eece-1a62-4928-b8da-4c54e6f01153-kube-api-access-qdxqr\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:11.910909 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.910732 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a03eece-1a62-4928-b8da-4c54e6f01153-root\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:11.910909 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.910757 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-wtmp\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:11.910909 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.910804 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a03eece-1a62-4928-b8da-4c54e6f01153-metrics-client-ca\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:11.910909 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.910847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a03eece-1a62-4928-b8da-4c54e6f01153-sys\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:11.910909 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.910872 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-accelerators-collector-config\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:11.910909 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.910894 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-textfile\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:11.911117 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.910991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:11.923722 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:11.923696 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f7499fb-57aa-4c9f-a48c-45252cd451f1" path="/var/lib/kubelet/pods/9f7499fb-57aa-4c9f-a48c-45252cd451f1/volumes" Jul 21 12:29:12.011883 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.011844 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-accelerators-collector-config\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.011883 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.011886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-textfile\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.012095 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.012020 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.012095 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.012057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-tls\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.012095 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.012079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdxqr\" (UniqueName: \"kubernetes.io/projected/0a03eece-1a62-4928-b8da-4c54e6f01153-kube-api-access-qdxqr\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.012230 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.012097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a03eece-1a62-4928-b8da-4c54e6f01153-root\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.012230 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.012129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-wtmp\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.012230 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.012159 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a03eece-1a62-4928-b8da-4c54e6f01153-metrics-client-ca\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.012230 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.012187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a03eece-1a62-4928-b8da-4c54e6f01153-sys\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.012230 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:29:12.012192 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 21 12:29:12.012230 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.012221 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a03eece-1a62-4928-b8da-4c54e6f01153-root\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.012502 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:29:12.012260 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-tls podName:0a03eece-1a62-4928-b8da-4c54e6f01153 nodeName:}" failed. No retries permitted until 2026-07-21 12:29:12.512238656 +0000 UTC m=+99.162652001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-tls") pod "node-exporter-qq577" (UID: "0a03eece-1a62-4928-b8da-4c54e6f01153") : secret "node-exporter-tls" not found Jul 21 12:29:12.012502 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.012350 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-wtmp\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.012502 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.012424 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a03eece-1a62-4928-b8da-4c54e6f01153-sys\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.012502 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.012428 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-textfile\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.012712 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.012571 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-accelerators-collector-config\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.012810 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.012790 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a03eece-1a62-4928-b8da-4c54e6f01153-metrics-client-ca\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.014385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.014353 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.020075 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.020052 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdxqr\" (UniqueName: \"kubernetes.io/projected/0a03eece-1a62-4928-b8da-4c54e6f01153-kube-api-access-qdxqr\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.517664 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.517631 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-tls\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.519989 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.519966 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a03eece-1a62-4928-b8da-4c54e6f01153-node-exporter-tls\") pod \"node-exporter-qq577\" (UID: \"0a03eece-1a62-4928-b8da-4c54e6f01153\") " pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.693648 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:12.693568 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qq577" Jul 21 12:29:12.707271 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:29:12.707241 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a03eece_1a62_4928_b8da_4c54e6f01153.slice/crio-a6187a45a57070801a30e61e0cd7e7a69ee48d494f1e42b3d30a158e6dbacb82 WatchSource:0}: Error finding container a6187a45a57070801a30e61e0cd7e7a69ee48d494f1e42b3d30a158e6dbacb82: Status 404 returned error can't find the container with id a6187a45a57070801a30e61e0cd7e7a69ee48d494f1e42b3d30a158e6dbacb82 Jul 21 12:29:13.433736 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:13.433694 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq577" event={"ID":"0a03eece-1a62-4928-b8da-4c54e6f01153","Type":"ContainerStarted","Data":"a6187a45a57070801a30e61e0cd7e7a69ee48d494f1e42b3d30a158e6dbacb82"} Jul 21 12:29:14.438851 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.438775 2574 generic.go:356] "Generic (PLEG): container finished" podID="0a03eece-1a62-4928-b8da-4c54e6f01153" containerID="d8fbc7b9f47550d033c357de28d89e00a9fcbe523cc82dfaea890c7d5476c050" exitCode=0 Jul 21 12:29:14.438851 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.438818 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq577" event={"ID":"0a03eece-1a62-4928-b8da-4c54e6f01153","Type":"ContainerDied","Data":"d8fbc7b9f47550d033c357de28d89e00a9fcbe523cc82dfaea890c7d5476c050"} Jul 21 12:29:14.725470 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.725396 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-75d76cc7d4-22llw"] Jul 21 12:29:14.749769 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.749743 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75d76cc7d4-22llw"] Jul 21 12:29:14.749904 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.749859 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:14.752711 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.752685 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-626wl\"" Jul 21 12:29:14.752866 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.752722 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 21 12:29:14.752866 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.752769 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 21 12:29:14.752866 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.752778 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 21 12:29:14.752866 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.752831 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 21 12:29:14.753133 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.753119 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 21 12:29:14.754018 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.753998 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ehafapthbarmh\"" Jul 21 12:29:14.935048 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.935018 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:14.935048 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.935052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv2hs\" (UniqueName: \"kubernetes.io/projected/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-kube-api-access-wv2hs\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:14.935247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.935076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-tls\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:14.935247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.935095 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:14.935247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.935184 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:14.935247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.935214 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-metrics-client-ca\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:14.935372 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.935258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-grpc-tls\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:14.935372 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:14.935279 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.035962 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.035912 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.036186 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.035969 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2hs\" (UniqueName: \"kubernetes.io/projected/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-kube-api-access-wv2hs\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.036186 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.036005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-tls\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.036186 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.036035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.036366 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.036189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.036366 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.036233 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-metrics-client-ca\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.036366 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.036294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-grpc-tls\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.036366 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.036341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.037402 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.037353 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-metrics-client-ca\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.038799 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.038779 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.039089 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.039060 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-tls\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.039171 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.039092 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.039210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.039185 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.039279 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.039257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-grpc-tls\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.039359 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.039345 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.044663 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.044642 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv2hs\" (UniqueName: \"kubernetes.io/projected/be3d825d-876a-4d3e-ab9e-5a988f92b3dd-kube-api-access-wv2hs\") pod \"thanos-querier-75d76cc7d4-22llw\" (UID: \"be3d825d-876a-4d3e-ab9e-5a988f92b3dd\") " pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.059526 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.059506 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:15.208490 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.208456 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75d76cc7d4-22llw"] Jul 21 12:29:15.211744 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:29:15.211714 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3d825d_876a_4d3e_ab9e_5a988f92b3dd.slice/crio-36b6b9cf5e7484c6314c42fe0ef54f3a016c6c1def1478226000f2c77e9dca2c WatchSource:0}: Error finding container 36b6b9cf5e7484c6314c42fe0ef54f3a016c6c1def1478226000f2c77e9dca2c: Status 404 returned error can't find the container with id 36b6b9cf5e7484c6314c42fe0ef54f3a016c6c1def1478226000f2c77e9dca2c Jul 21 12:29:15.443857 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.443760 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" event={"ID":"be3d825d-876a-4d3e-ab9e-5a988f92b3dd","Type":"ContainerStarted","Data":"36b6b9cf5e7484c6314c42fe0ef54f3a016c6c1def1478226000f2c77e9dca2c"} Jul 21 12:29:15.445865 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.445841 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq577" event={"ID":"0a03eece-1a62-4928-b8da-4c54e6f01153","Type":"ContainerStarted","Data":"7a7886fdd48f911f1f378e850e5dfd6baec077f2adfc88f14f527b965594de68"} Jul 21 12:29:15.446009 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.445889 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq577" event={"ID":"0a03eece-1a62-4928-b8da-4c54e6f01153","Type":"ContainerStarted","Data":"c3b336c404ba0c36006bbcf38d0b0f0180ec670a89bf5558065d4346d4b261b2"} Jul 21 12:29:15.465137 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:15.465091 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qq577" podStartSLOduration=3.02440296 podStartE2EDuration="4.465075959s" totalImagesPullingTime="1.369475891s" totalInitContainerRuntime="71.197108ms" isStatefulPod=true podCreationTimestamp="2026-07-21 12:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:29:15.46403113 +0000 UTC m=+102.114444492" watchObservedRunningTime="2026-07-21 12:29:15.465075959 +0000 UTC m=+102.115489366" Jul 21 12:29:16.550787 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.550754 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr"] Jul 21 12:29:16.585072 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.585043 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr" Jul 21 12:29:16.589714 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.589668 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jnbmd\"" Jul 21 12:29:16.589714 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.589703 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 21 12:29:16.590627 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.590597 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr"] Jul 21 12:29:16.749089 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.749052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8868a4a9-c37b-45dd-b569-11b1dde7848f-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-rcqcr\" (UID: \"8868a4a9-c37b-45dd-b569-11b1dde7848f\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr" Jul 21 12:29:16.749251 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.749181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j2gz\" (UniqueName: \"kubernetes.io/projected/8868a4a9-c37b-45dd-b569-11b1dde7848f-kube-api-access-8j2gz\") pod \"monitoring-plugin-7b7b7bc8bf-rcqcr\" (UID: \"8868a4a9-c37b-45dd-b569-11b1dde7848f\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr" Jul 21 12:29:16.850102 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.849930 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j2gz\" (UniqueName: \"kubernetes.io/projected/8868a4a9-c37b-45dd-b569-11b1dde7848f-kube-api-access-8j2gz\") pod \"monitoring-plugin-7b7b7bc8bf-rcqcr\" (UID: \"8868a4a9-c37b-45dd-b569-11b1dde7848f\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr" Jul 21 12:29:16.850102 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.850013 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8868a4a9-c37b-45dd-b569-11b1dde7848f-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-rcqcr\" (UID: \"8868a4a9-c37b-45dd-b569-11b1dde7848f\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr" Jul 21 12:29:16.854493 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.854431 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8868a4a9-c37b-45dd-b569-11b1dde7848f-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-rcqcr\" (UID: \"8868a4a9-c37b-45dd-b569-11b1dde7848f\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr" Jul 21 12:29:16.858462 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.858436 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j2gz\" (UniqueName: \"kubernetes.io/projected/8868a4a9-c37b-45dd-b569-11b1dde7848f-kube-api-access-8j2gz\") pod \"monitoring-plugin-7b7b7bc8bf-rcqcr\" (UID: \"8868a4a9-c37b-45dd-b569-11b1dde7848f\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr" Jul 21 12:29:16.897385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.897354 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr" Jul 21 12:29:16.950659 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.950629 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-865947956c-gb2lf"] Jul 21 12:29:16.971700 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.971660 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-865947956c-gb2lf"] Jul 21 12:29:16.971869 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.971799 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:16.974763 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.974733 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xzbdb\"" Jul 21 12:29:16.974911 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.974740 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 21 12:29:16.974911 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.974740 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 21 12:29:16.975092 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.974968 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 21 12:29:16.975224 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.975187 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 21 12:29:16.975224 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.975213 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 21 12:29:16.981725 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:16.981352 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 21 12:29:17.153516 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.153434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f35cbae9-4a0a-4250-8a0c-e205231fc564-federate-client-tls\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.153516 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.153492 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f35cbae9-4a0a-4250-8a0c-e205231fc564-telemeter-trusted-ca-bundle\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.153744 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.153525 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f35cbae9-4a0a-4250-8a0c-e205231fc564-serving-certs-ca-bundle\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.153744 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.153551 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f35cbae9-4a0a-4250-8a0c-e205231fc564-telemeter-client-tls\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.153744 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.153583 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf858\" (UniqueName: \"kubernetes.io/projected/f35cbae9-4a0a-4250-8a0c-e205231fc564-kube-api-access-tf858\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.153744 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.153636 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f35cbae9-4a0a-4250-8a0c-e205231fc564-metrics-client-ca\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.153744 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.153668 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f35cbae9-4a0a-4250-8a0c-e205231fc564-secret-telemeter-client\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.153744 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.153730 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f35cbae9-4a0a-4250-8a0c-e205231fc564-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.254735 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.254700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f35cbae9-4a0a-4250-8a0c-e205231fc564-secret-telemeter-client\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.254919 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.254792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f35cbae9-4a0a-4250-8a0c-e205231fc564-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.254919 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.254832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f35cbae9-4a0a-4250-8a0c-e205231fc564-federate-client-tls\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.254919 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.254880 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f35cbae9-4a0a-4250-8a0c-e205231fc564-telemeter-trusted-ca-bundle\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.254919 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.254911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f35cbae9-4a0a-4250-8a0c-e205231fc564-serving-certs-ca-bundle\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.255097 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.254961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f35cbae9-4a0a-4250-8a0c-e205231fc564-telemeter-client-tls\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.255097 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.255019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf858\" (UniqueName: \"kubernetes.io/projected/f35cbae9-4a0a-4250-8a0c-e205231fc564-kube-api-access-tf858\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.255097 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.255057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f35cbae9-4a0a-4250-8a0c-e205231fc564-metrics-client-ca\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.256964 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.256852 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f35cbae9-4a0a-4250-8a0c-e205231fc564-metrics-client-ca\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.257962 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.257446 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f35cbae9-4a0a-4250-8a0c-e205231fc564-serving-certs-ca-bundle\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.257962 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.257884 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f35cbae9-4a0a-4250-8a0c-e205231fc564-telemeter-trusted-ca-bundle\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.257962 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.257956 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f35cbae9-4a0a-4250-8a0c-e205231fc564-secret-telemeter-client\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.259623 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.259239 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f35cbae9-4a0a-4250-8a0c-e205231fc564-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.260133 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.259855 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f35cbae9-4a0a-4250-8a0c-e205231fc564-telemeter-client-tls\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.263601 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.263570 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f35cbae9-4a0a-4250-8a0c-e205231fc564-federate-client-tls\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.268133 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.268108 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf858\" (UniqueName: \"kubernetes.io/projected/f35cbae9-4a0a-4250-8a0c-e205231fc564-kube-api-access-tf858\") pod \"telemeter-client-865947956c-gb2lf\" (UID: \"f35cbae9-4a0a-4250-8a0c-e205231fc564\") " pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.284844 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.284819 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" Jul 21 12:29:17.508323 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.508298 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-865947956c-gb2lf"] Jul 21 12:29:17.520211 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:17.520187 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr"] Jul 21 12:29:17.564470 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:29:17.564437 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35cbae9_4a0a_4250_8a0c_e205231fc564.slice/crio-1af0b2b02cadfaeb24dddaa606f932502a4b29192da8e87e58fb08233c9a5f30 WatchSource:0}: Error finding container 1af0b2b02cadfaeb24dddaa606f932502a4b29192da8e87e58fb08233c9a5f30: Status 404 returned error can't find the container with id 1af0b2b02cadfaeb24dddaa606f932502a4b29192da8e87e58fb08233c9a5f30 Jul 21 12:29:17.565109 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:29:17.565080 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8868a4a9_c37b_45dd_b569_11b1dde7848f.slice/crio-16e48a9754e077f94dd3fce678ef1bb8d5f76a2c0f6940552dc3862bbb495f9d WatchSource:0}: Error finding container 16e48a9754e077f94dd3fce678ef1bb8d5f76a2c0f6940552dc3862bbb495f9d: Status 404 returned error can't find the container with id 16e48a9754e077f94dd3fce678ef1bb8d5f76a2c0f6940552dc3862bbb495f9d Jul 21 12:29:18.461282 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:18.461244 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr" event={"ID":"8868a4a9-c37b-45dd-b569-11b1dde7848f","Type":"ContainerStarted","Data":"16e48a9754e077f94dd3fce678ef1bb8d5f76a2c0f6940552dc3862bbb495f9d"} Jul 21 12:29:18.463775 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:18.463743 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" event={"ID":"be3d825d-876a-4d3e-ab9e-5a988f92b3dd","Type":"ContainerStarted","Data":"de35871b5a410c0cec33d0928b1a6ccf70d9d3baf4b12982a6d21d769c5d60bc"} Jul 21 12:29:18.463960 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:18.463826 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" event={"ID":"be3d825d-876a-4d3e-ab9e-5a988f92b3dd","Type":"ContainerStarted","Data":"70714a56bea65bb781d938eeafb19ef34f0bf9b803b5ee02979baaf2ba238c2f"} Jul 21 12:29:18.463960 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:18.463869 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" event={"ID":"be3d825d-876a-4d3e-ab9e-5a988f92b3dd","Type":"ContainerStarted","Data":"35357ef59ecc25f1a05b63c652abffc49f88661e28d5cf883dd61a5ab2ba6086"} Jul 21 12:29:18.465045 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:18.465006 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" event={"ID":"f35cbae9-4a0a-4250-8a0c-e205231fc564","Type":"ContainerStarted","Data":"1af0b2b02cadfaeb24dddaa606f932502a4b29192da8e87e58fb08233c9a5f30"} Jul 21 12:29:20.334325 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:20.334297 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xjxkh" Jul 21 12:29:21.481673 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:21.481626 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" event={"ID":"be3d825d-876a-4d3e-ab9e-5a988f92b3dd","Type":"ContainerStarted","Data":"05fc98eed3ffcbc18e26e24ebf1fe7698812590bed83fc1286c8fc42980bb4c3"} Jul 21 12:29:21.482159 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:21.481714 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" event={"ID":"be3d825d-876a-4d3e-ab9e-5a988f92b3dd","Type":"ContainerStarted","Data":"3f7dfc58f7dcea6528a4b6179220b328e77ffcbdacf97d4b21fe33f5d0c38216"} Jul 21 12:29:21.482159 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:21.481756 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" event={"ID":"be3d825d-876a-4d3e-ab9e-5a988f92b3dd","Type":"ContainerStarted","Data":"ee0a92ef3e5335fdc08b6030d68c067c12a25879862ba7b0e8415300d436cbd5"} Jul 21 12:29:21.482159 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:21.481796 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:21.483509 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:21.483470 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" event={"ID":"f35cbae9-4a0a-4250-8a0c-e205231fc564","Type":"ContainerStarted","Data":"ba24b3ffee621f8ed66c75e7551d0c27bbfe34d7ae2986678351f3cb84f0abd3"} Jul 21 12:29:21.485146 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:21.485119 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr" event={"ID":"8868a4a9-c37b-45dd-b569-11b1dde7848f","Type":"ContainerStarted","Data":"ecd68d0916773d258e9fdcf2a91c347c331e4b68643822928c0e4bf5351a254c"} Jul 21 12:29:21.485255 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:21.485223 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr" Jul 21 12:29:21.490583 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:21.490562 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr" Jul 21 12:29:21.506868 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:21.506815 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" podStartSLOduration=2.5232669789999997 podStartE2EDuration="7.506794946s" totalImagesPullingTime="4.983527967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:29:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:29:21.503691315 +0000 UTC m=+108.154104678" watchObservedRunningTime="2026-07-21 12:29:21.506794946 +0000 UTC m=+108.157208308" Jul 21 12:29:21.520014 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:21.519966 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-rcqcr" podStartSLOduration=2.554940639 podStartE2EDuration="5.519956897s" totalImagesPullingTime="2.965016258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:29:21.519108684 +0000 UTC m=+108.169522047" watchObservedRunningTime="2026-07-21 12:29:21.519956897 +0000 UTC m=+108.170370251" Jul 21 12:29:22.492064 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:22.491977 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" event={"ID":"f35cbae9-4a0a-4250-8a0c-e205231fc564","Type":"ContainerStarted","Data":"afd80a758ce4201ecfe7d1aa3c9d50e7978ae0ce4565a77dd5648b307ea3bea4"} Jul 21 12:29:22.492064 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:22.492032 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" event={"ID":"f35cbae9-4a0a-4250-8a0c-e205231fc564","Type":"ContainerStarted","Data":"f6e3b0b65f8219e4b5aec6e5a809c448a6cc7e721b953e2a6b5172b9d4974b62"} Jul 21 12:29:22.497665 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:22.497642 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-75d76cc7d4-22llw" Jul 21 12:29:22.516752 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:22.516706 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-865947956c-gb2lf" podStartSLOduration=2.140832566 podStartE2EDuration="6.516690851s" totalImagesPullingTime="4.375858285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:29:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:29:22.514086044 +0000 UTC m=+109.164499430" watchObservedRunningTime="2026-07-21 12:29:22.516690851 +0000 UTC m=+109.167104211" Jul 21 12:29:29.520682 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:29.520650 2574 generic.go:356] "Generic (PLEG): container finished" podID="8947a59a-697b-41e4-9ecc-dfd60f9eab33" containerID="e81582eac78ddcc8945b9a7594ad2646f169beb132a767dd6d16c0bc03a45780" exitCode=0 Jul 21 12:29:29.521123 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:29.520732 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" event={"ID":"8947a59a-697b-41e4-9ecc-dfd60f9eab33","Type":"ContainerDied","Data":"e81582eac78ddcc8945b9a7594ad2646f169beb132a767dd6d16c0bc03a45780"} Jul 21 12:29:29.521123 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:29.521090 2574 scope.go:117] "RemoveContainer" containerID="e81582eac78ddcc8945b9a7594ad2646f169beb132a767dd6d16c0bc03a45780" Jul 21 12:29:30.526201 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:30.526162 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-w5n22" event={"ID":"8947a59a-697b-41e4-9ecc-dfd60f9eab33","Type":"ContainerStarted","Data":"8e8a4c90ebdc34303eee8d7df865f4782a32b7d724705dedfc7fdfd2355c2c08"} Jul 21 12:29:34.188312 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:29:34.188230 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d964404_b5ad_45f6_90c0_e70bb3986442.slice/crio-3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5.scope\": RecentStats: unable to find data in memory cache]" Jul 21 12:29:34.542803 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:34.542770 2574 generic.go:356] "Generic (PLEG): container finished" podID="4d964404-b5ad-45f6-90c0-e70bb3986442" containerID="3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5" exitCode=0 Jul 21 12:29:34.542979 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:34.542824 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-59v4g" event={"ID":"4d964404-b5ad-45f6-90c0-e70bb3986442","Type":"ContainerDied","Data":"3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5"} Jul 21 12:29:34.543211 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:34.543194 2574 scope.go:117] "RemoveContainer" containerID="3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5" Jul 21 12:29:35.451640 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:29:35.451607 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d964404_b5ad_45f6_90c0_e70bb3986442.slice/crio-3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5.scope\": RecentStats: unable to find data in memory cache]" Jul 21 12:29:35.549466 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:35.549434 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-59v4g" event={"ID":"4d964404-b5ad-45f6-90c0-e70bb3986442","Type":"ContainerStarted","Data":"39930dcb978ffb71fc7a1cec388fbf2301c82567181d69427fa158d2d372a310"} Jul 21 12:29:39.565667 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:39.565631 2574 generic.go:356] "Generic (PLEG): container finished" podID="9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7" containerID="9efe10e84fea604ba07302f864c2abd2f77e18ff80f65547f71154b519b52bdb" exitCode=0 Jul 21 12:29:39.566074 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:39.565708 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" event={"ID":"9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7","Type":"ContainerDied","Data":"9efe10e84fea604ba07302f864c2abd2f77e18ff80f65547f71154b519b52bdb"} Jul 21 12:29:39.566116 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:39.566088 2574 scope.go:117] "RemoveContainer" containerID="9efe10e84fea604ba07302f864c2abd2f77e18ff80f65547f71154b519b52bdb" Jul 21 12:29:40.571192 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:29:40.571159 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-z7shn" event={"ID":"9fd3ae9a-d294-4c3d-a925-7f3e726f4ee7","Type":"ContainerStarted","Data":"fa86a2e6ce33feed8871ace2b35f8ec8eb597f286965cbd48793609fa242736e"} Jul 21 12:29:44.226233 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:29:44.226194 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d964404_b5ad_45f6_90c0_e70bb3986442.slice/crio-3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5.scope\": RecentStats: unable to find data in memory cache]" Jul 21 12:29:50.490239 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:29:50.490206 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d964404_b5ad_45f6_90c0_e70bb3986442.slice/crio-3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5.scope\": RecentStats: unable to find data in memory cache]" Jul 21 12:29:54.234885 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:29:54.234851 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d964404_b5ad_45f6_90c0_e70bb3986442.slice/crio-3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5.scope\": RecentStats: unable to find data in memory cache]" Jul 21 12:30:04.242635 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:30:04.242596 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d964404_b5ad_45f6_90c0_e70bb3986442.slice/crio-3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5.scope\": RecentStats: unable to find data in memory cache]" Jul 21 12:30:05.467738 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:30:05.467699 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d964404_b5ad_45f6_90c0_e70bb3986442.slice/crio-3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5.scope\": RecentStats: unable to find data in memory cache]" Jul 21 12:30:14.280545 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:30:14.280513 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d964404_b5ad_45f6_90c0_e70bb3986442.slice/crio-3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5.scope\": RecentStats: unable to find data in memory cache]" Jul 21 12:30:20.492779 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:30:20.492688 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d964404_b5ad_45f6_90c0_e70bb3986442.slice/crio-3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5.scope\": RecentStats: unable to find data in memory cache]" Jul 21 12:30:24.283279 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:30:24.283241 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d964404_b5ad_45f6_90c0_e70bb3986442.slice/crio-3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5.scope\": RecentStats: unable to find data in memory cache]" Jul 21 12:30:24.283742 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:30:24.283290 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d964404_b5ad_45f6_90c0_e70bb3986442.slice/crio-3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5.scope\": RecentStats: unable to find data in memory cache]" Jul 21 12:30:24.287995 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:30:24.287969 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d964404_b5ad_45f6_90c0_e70bb3986442.slice/crio-3bb74663e4efa6ace2243f2a1873490897a78796b8dfd1e14649068191401da5.scope\": RecentStats: unable to find data in memory cache]" Jul 21 12:31:29.107315 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.107282 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn"] Jul 21 12:31:29.110502 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.110486 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" Jul 21 12:31:29.113364 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.113344 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 12:31:29.113492 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.113388 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wpjd9\"" Jul 21 12:31:29.114465 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.114446 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 12:31:29.116092 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.116066 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9lz4\" (UniqueName: \"kubernetes.io/projected/00b33ff9-2a54-4573-80a5-92edc201856c-kube-api-access-f9lz4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn\" (UID: \"00b33ff9-2a54-4573-80a5-92edc201856c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" Jul 21 12:31:29.116178 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.116111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00b33ff9-2a54-4573-80a5-92edc201856c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn\" (UID: \"00b33ff9-2a54-4573-80a5-92edc201856c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" Jul 21 12:31:29.116178 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.116136 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00b33ff9-2a54-4573-80a5-92edc201856c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn\" (UID: \"00b33ff9-2a54-4573-80a5-92edc201856c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" Jul 21 12:31:29.127226 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.127192 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn"] Jul 21 12:31:29.216723 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.216685 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9lz4\" (UniqueName: \"kubernetes.io/projected/00b33ff9-2a54-4573-80a5-92edc201856c-kube-api-access-f9lz4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn\" (UID: \"00b33ff9-2a54-4573-80a5-92edc201856c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" Jul 21 12:31:29.216905 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.216735 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00b33ff9-2a54-4573-80a5-92edc201856c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn\" (UID: \"00b33ff9-2a54-4573-80a5-92edc201856c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" Jul 21 12:31:29.216905 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.216758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00b33ff9-2a54-4573-80a5-92edc201856c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn\" (UID: \"00b33ff9-2a54-4573-80a5-92edc201856c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" Jul 21 12:31:29.217128 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.217112 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00b33ff9-2a54-4573-80a5-92edc201856c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn\" (UID: \"00b33ff9-2a54-4573-80a5-92edc201856c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" Jul 21 12:31:29.217167 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.217150 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00b33ff9-2a54-4573-80a5-92edc201856c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn\" (UID: \"00b33ff9-2a54-4573-80a5-92edc201856c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" Jul 21 12:31:29.226485 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.226459 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9lz4\" (UniqueName: \"kubernetes.io/projected/00b33ff9-2a54-4573-80a5-92edc201856c-kube-api-access-f9lz4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn\" (UID: \"00b33ff9-2a54-4573-80a5-92edc201856c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" Jul 21 12:31:29.419688 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.419589 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" Jul 21 12:31:29.556365 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.556334 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn"] Jul 21 12:31:29.560035 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:31:29.560005 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b33ff9_2a54_4573_80a5_92edc201856c.slice/crio-190b082c1973dbdf782a31a5a63df011198fa227fdb5c6210c25bb5a03617c1f WatchSource:0}: Error finding container 190b082c1973dbdf782a31a5a63df011198fa227fdb5c6210c25bb5a03617c1f: Status 404 returned error can't find the container with id 190b082c1973dbdf782a31a5a63df011198fa227fdb5c6210c25bb5a03617c1f Jul 21 12:31:29.979548 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:29.979519 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" event={"ID":"00b33ff9-2a54-4573-80a5-92edc201856c","Type":"ContainerStarted","Data":"190b082c1973dbdf782a31a5a63df011198fa227fdb5c6210c25bb5a03617c1f"} Jul 21 12:31:35.001317 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:35.001279 2574 generic.go:356] "Generic (PLEG): container finished" podID="00b33ff9-2a54-4573-80a5-92edc201856c" containerID="88390db77410e12d7677365550607ad6bd7cd8010403b68fccae65e88bd1a15a" exitCode=0 Jul 21 12:31:35.001695 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:35.001363 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" event={"ID":"00b33ff9-2a54-4573-80a5-92edc201856c","Type":"ContainerDied","Data":"88390db77410e12d7677365550607ad6bd7cd8010403b68fccae65e88bd1a15a"} Jul 21 12:31:37.012382 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:37.012353 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" event={"ID":"00b33ff9-2a54-4573-80a5-92edc201856c","Type":"ContainerStarted","Data":"1625cd4f9ff5d359dacf3ba338221d26d799e37043782b30b25afa87c80325a3"} Jul 21 12:31:38.017734 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:38.017702 2574 generic.go:356] "Generic (PLEG): container finished" podID="00b33ff9-2a54-4573-80a5-92edc201856c" containerID="1625cd4f9ff5d359dacf3ba338221d26d799e37043782b30b25afa87c80325a3" exitCode=0 Jul 21 12:31:38.018152 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:38.017786 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" event={"ID":"00b33ff9-2a54-4573-80a5-92edc201856c","Type":"ContainerDied","Data":"1625cd4f9ff5d359dacf3ba338221d26d799e37043782b30b25afa87c80325a3"} Jul 21 12:31:47.058472 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:47.058437 2574 generic.go:356] "Generic (PLEG): container finished" podID="00b33ff9-2a54-4573-80a5-92edc201856c" containerID="7913ceb407188bb4d5c53771e610210481ce3eaeec05838f0ed943589b302921" exitCode=0 Jul 21 12:31:47.058874 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:47.058496 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" event={"ID":"00b33ff9-2a54-4573-80a5-92edc201856c","Type":"ContainerDied","Data":"7913ceb407188bb4d5c53771e610210481ce3eaeec05838f0ed943589b302921"} Jul 21 12:31:48.189785 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:48.189760 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" Jul 21 12:31:48.277042 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:48.277001 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9lz4\" (UniqueName: \"kubernetes.io/projected/00b33ff9-2a54-4573-80a5-92edc201856c-kube-api-access-f9lz4\") pod \"00b33ff9-2a54-4573-80a5-92edc201856c\" (UID: \"00b33ff9-2a54-4573-80a5-92edc201856c\") " Jul 21 12:31:48.277239 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:48.277083 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00b33ff9-2a54-4573-80a5-92edc201856c-bundle\") pod \"00b33ff9-2a54-4573-80a5-92edc201856c\" (UID: \"00b33ff9-2a54-4573-80a5-92edc201856c\") " Jul 21 12:31:48.277239 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:48.277141 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00b33ff9-2a54-4573-80a5-92edc201856c-util\") pod \"00b33ff9-2a54-4573-80a5-92edc201856c\" (UID: \"00b33ff9-2a54-4573-80a5-92edc201856c\") " Jul 21 12:31:48.277742 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:48.277707 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b33ff9-2a54-4573-80a5-92edc201856c-bundle" (OuterVolumeSpecName: "bundle") pod "00b33ff9-2a54-4573-80a5-92edc201856c" (UID: "00b33ff9-2a54-4573-80a5-92edc201856c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:31:48.279207 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:48.279178 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b33ff9-2a54-4573-80a5-92edc201856c-kube-api-access-f9lz4" (OuterVolumeSpecName: "kube-api-access-f9lz4") pod "00b33ff9-2a54-4573-80a5-92edc201856c" (UID: "00b33ff9-2a54-4573-80a5-92edc201856c"). InnerVolumeSpecName "kube-api-access-f9lz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:31:48.282872 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:48.282841 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b33ff9-2a54-4573-80a5-92edc201856c-util" (OuterVolumeSpecName: "util") pod "00b33ff9-2a54-4573-80a5-92edc201856c" (UID: "00b33ff9-2a54-4573-80a5-92edc201856c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:31:48.377779 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:48.377686 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9lz4\" (UniqueName: \"kubernetes.io/projected/00b33ff9-2a54-4573-80a5-92edc201856c-kube-api-access-f9lz4\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:31:48.377779 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:48.377718 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00b33ff9-2a54-4573-80a5-92edc201856c-bundle\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:31:48.377779 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:48.377728 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00b33ff9-2a54-4573-80a5-92edc201856c-util\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:31:49.068273 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:49.068234 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" event={"ID":"00b33ff9-2a54-4573-80a5-92edc201856c","Type":"ContainerDied","Data":"190b082c1973dbdf782a31a5a63df011198fa227fdb5c6210c25bb5a03617c1f"} Jul 21 12:31:49.068440 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:49.068280 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r26cn" Jul 21 12:31:49.068440 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:49.068286 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="190b082c1973dbdf782a31a5a63df011198fa227fdb5c6210c25bb5a03617c1f" Jul 21 12:31:52.185220 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.185182 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nmtg8"] Jul 21 12:31:52.185690 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.185603 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b33ff9-2a54-4573-80a5-92edc201856c" containerName="extract" Jul 21 12:31:52.185690 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.185621 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b33ff9-2a54-4573-80a5-92edc201856c" containerName="extract" Jul 21 12:31:52.185690 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.185654 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b33ff9-2a54-4573-80a5-92edc201856c" containerName="util" Jul 21 12:31:52.185690 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.185663 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b33ff9-2a54-4573-80a5-92edc201856c" containerName="util" Jul 21 12:31:52.185690 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.185673 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b33ff9-2a54-4573-80a5-92edc201856c" containerName="pull" Jul 21 12:31:52.185690 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.185681 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b33ff9-2a54-4573-80a5-92edc201856c" containerName="pull" Jul 21 12:31:52.186065 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.185761 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="00b33ff9-2a54-4573-80a5-92edc201856c" containerName="extract" Jul 21 12:31:52.230103 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.230067 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nmtg8"] Jul 21 12:31:52.230262 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.230190 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nmtg8" Jul 21 12:31:52.233075 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.233047 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 21 12:31:52.233216 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.233199 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-pk2m9\"" Jul 21 12:31:52.233300 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.233284 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 21 12:31:52.312002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.311933 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5723a72f-79bd-409f-b66e-3662fd566959-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nmtg8\" (UID: \"5723a72f-79bd-409f-b66e-3662fd566959\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nmtg8" Jul 21 12:31:52.312200 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.312022 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2jmw\" (UniqueName: \"kubernetes.io/projected/5723a72f-79bd-409f-b66e-3662fd566959-kube-api-access-r2jmw\") pod \"cert-manager-operator-controller-manager-55c444bf86-nmtg8\" (UID: \"5723a72f-79bd-409f-b66e-3662fd566959\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nmtg8" Jul 21 12:31:52.413510 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.413468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5723a72f-79bd-409f-b66e-3662fd566959-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nmtg8\" (UID: \"5723a72f-79bd-409f-b66e-3662fd566959\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nmtg8" Jul 21 12:31:52.413685 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.413517 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2jmw\" (UniqueName: \"kubernetes.io/projected/5723a72f-79bd-409f-b66e-3662fd566959-kube-api-access-r2jmw\") pod \"cert-manager-operator-controller-manager-55c444bf86-nmtg8\" (UID: \"5723a72f-79bd-409f-b66e-3662fd566959\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nmtg8" Jul 21 12:31:52.413924 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.413898 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5723a72f-79bd-409f-b66e-3662fd566959-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nmtg8\" (UID: \"5723a72f-79bd-409f-b66e-3662fd566959\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nmtg8" Jul 21 12:31:52.423194 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.423168 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2jmw\" (UniqueName: \"kubernetes.io/projected/5723a72f-79bd-409f-b66e-3662fd566959-kube-api-access-r2jmw\") pod \"cert-manager-operator-controller-manager-55c444bf86-nmtg8\" (UID: \"5723a72f-79bd-409f-b66e-3662fd566959\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nmtg8" Jul 21 12:31:52.539278 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.539243 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nmtg8" Jul 21 12:31:52.695519 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:52.695488 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nmtg8"] Jul 21 12:31:52.698550 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:31:52.698524 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5723a72f_79bd_409f_b66e_3662fd566959.slice/crio-b14f5540753223fdf83c6e8e423a341f2fd380fef6ba4c8cb1e007713d38bd34 WatchSource:0}: Error finding container b14f5540753223fdf83c6e8e423a341f2fd380fef6ba4c8cb1e007713d38bd34: Status 404 returned error can't find the container with id b14f5540753223fdf83c6e8e423a341f2fd380fef6ba4c8cb1e007713d38bd34 Jul 21 12:31:53.085002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:53.084964 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nmtg8" event={"ID":"5723a72f-79bd-409f-b66e-3662fd566959","Type":"ContainerStarted","Data":"b14f5540753223fdf83c6e8e423a341f2fd380fef6ba4c8cb1e007713d38bd34"} Jul 21 12:31:55.095295 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.095260 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nmtg8" event={"ID":"5723a72f-79bd-409f-b66e-3662fd566959","Type":"ContainerStarted","Data":"56e3f1a90984857ed42dc0ed19d8bffaa30a6ec9bde4889fcc10c3d1467e43e2"} Jul 21 12:31:55.116780 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.116726 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nmtg8" podStartSLOduration=1.18343031 podStartE2EDuration="3.116709174s" totalImagesPullingTime="1.933278864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:31:55.114640922 +0000 UTC m=+261.765054284" watchObservedRunningTime="2026-07-21 12:31:55.116709174 +0000 UTC m=+261.767122535" Jul 21 12:31:55.547098 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.547055 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-g9l2p"] Jul 21 12:31:55.549611 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.549585 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-g9l2p" Jul 21 12:31:55.552312 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.552285 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 12:31:55.553377 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.553355 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-89gj2\"" Jul 21 12:31:55.553499 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.553382 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 12:31:55.565050 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.565019 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-g9l2p"] Jul 21 12:31:55.643742 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.643707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52e48d1e-6522-49a3-9c4f-655a439d2057-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-g9l2p\" (UID: \"52e48d1e-6522-49a3-9c4f-655a439d2057\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g9l2p" Jul 21 12:31:55.643972 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.643774 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbs5m\" (UniqueName: \"kubernetes.io/projected/52e48d1e-6522-49a3-9c4f-655a439d2057-kube-api-access-xbs5m\") pod \"cert-manager-webhook-6486ffd997-g9l2p\" (UID: \"52e48d1e-6522-49a3-9c4f-655a439d2057\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g9l2p" Jul 21 12:31:55.744998 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.744961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbs5m\" (UniqueName: \"kubernetes.io/projected/52e48d1e-6522-49a3-9c4f-655a439d2057-kube-api-access-xbs5m\") pod \"cert-manager-webhook-6486ffd997-g9l2p\" (UID: \"52e48d1e-6522-49a3-9c4f-655a439d2057\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g9l2p" Jul 21 12:31:55.745186 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.745066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52e48d1e-6522-49a3-9c4f-655a439d2057-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-g9l2p\" (UID: \"52e48d1e-6522-49a3-9c4f-655a439d2057\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g9l2p" Jul 21 12:31:55.763365 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.763335 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbs5m\" (UniqueName: \"kubernetes.io/projected/52e48d1e-6522-49a3-9c4f-655a439d2057-kube-api-access-xbs5m\") pod \"cert-manager-webhook-6486ffd997-g9l2p\" (UID: \"52e48d1e-6522-49a3-9c4f-655a439d2057\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g9l2p" Jul 21 12:31:55.764999 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.764963 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52e48d1e-6522-49a3-9c4f-655a439d2057-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-g9l2p\" (UID: \"52e48d1e-6522-49a3-9c4f-655a439d2057\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g9l2p" Jul 21 12:31:55.877769 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:55.877679 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-g9l2p" Jul 21 12:31:56.049644 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.049506 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-g9l2p"] Jul 21 12:31:56.052137 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:31:56.052111 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52e48d1e_6522_49a3_9c4f_655a439d2057.slice/crio-b5ffaab563a32597c6f622b81eef695d23fb5d9b9561b7388753078c61f8dd3f WatchSource:0}: Error finding container b5ffaab563a32597c6f622b81eef695d23fb5d9b9561b7388753078c61f8dd3f: Status 404 returned error can't find the container with id b5ffaab563a32597c6f622b81eef695d23fb5d9b9561b7388753078c61f8dd3f Jul 21 12:31:56.101238 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.101200 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-g9l2p" event={"ID":"52e48d1e-6522-49a3-9c4f-655a439d2057","Type":"ContainerStarted","Data":"b5ffaab563a32597c6f622b81eef695d23fb5d9b9561b7388753078c61f8dd3f"} Jul 21 12:31:56.391591 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.391553 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh"] Jul 21 12:31:56.394837 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.394813 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" Jul 21 12:31:56.399139 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.399114 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 12:31:56.407572 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.407551 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wpjd9\"" Jul 21 12:31:56.407710 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.407692 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 12:31:56.429581 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.429540 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh"] Jul 21 12:31:56.451687 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.451653 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2aef0038-decc-47ea-b002-b871ad11603f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh\" (UID: \"2aef0038-decc-47ea-b002-b871ad11603f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" Jul 21 12:31:56.451873 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.451695 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6prs\" (UniqueName: \"kubernetes.io/projected/2aef0038-decc-47ea-b002-b871ad11603f-kube-api-access-v6prs\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh\" (UID: \"2aef0038-decc-47ea-b002-b871ad11603f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" Jul 21 12:31:56.451873 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.451781 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aef0038-decc-47ea-b002-b871ad11603f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh\" (UID: \"2aef0038-decc-47ea-b002-b871ad11603f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" Jul 21 12:31:56.552789 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.552745 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2aef0038-decc-47ea-b002-b871ad11603f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh\" (UID: \"2aef0038-decc-47ea-b002-b871ad11603f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" Jul 21 12:31:56.553027 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.552799 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6prs\" (UniqueName: \"kubernetes.io/projected/2aef0038-decc-47ea-b002-b871ad11603f-kube-api-access-v6prs\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh\" (UID: \"2aef0038-decc-47ea-b002-b871ad11603f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" Jul 21 12:31:56.553027 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.552842 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aef0038-decc-47ea-b002-b871ad11603f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh\" (UID: \"2aef0038-decc-47ea-b002-b871ad11603f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" Jul 21 12:31:56.553191 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.553169 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2aef0038-decc-47ea-b002-b871ad11603f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh\" (UID: \"2aef0038-decc-47ea-b002-b871ad11603f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" Jul 21 12:31:56.553241 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.553199 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aef0038-decc-47ea-b002-b871ad11603f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh\" (UID: \"2aef0038-decc-47ea-b002-b871ad11603f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" Jul 21 12:31:56.562749 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.562716 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6prs\" (UniqueName: \"kubernetes.io/projected/2aef0038-decc-47ea-b002-b871ad11603f-kube-api-access-v6prs\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh\" (UID: \"2aef0038-decc-47ea-b002-b871ad11603f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" Jul 21 12:31:56.704787 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.704686 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" Jul 21 12:31:56.863496 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:56.863463 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh"] Jul 21 12:31:56.865406 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:31:56.865373 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aef0038_decc_47ea_b002_b871ad11603f.slice/crio-5d7872f327b988be65a4a8b9694c26acf6f7ba26e816697816759a3f9b07c055 WatchSource:0}: Error finding container 5d7872f327b988be65a4a8b9694c26acf6f7ba26e816697816759a3f9b07c055: Status 404 returned error can't find the container with id 5d7872f327b988be65a4a8b9694c26acf6f7ba26e816697816759a3f9b07c055 Jul 21 12:31:57.108496 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:57.108463 2574 generic.go:356] "Generic (PLEG): container finished" podID="2aef0038-decc-47ea-b002-b871ad11603f" containerID="7e6914fb8e5c3f9dd748d6ef1b020f2c92dfec461d82ad7ffdca07fb89a23b00" exitCode=0 Jul 21 12:31:57.108908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:57.108503 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" event={"ID":"2aef0038-decc-47ea-b002-b871ad11603f","Type":"ContainerDied","Data":"7e6914fb8e5c3f9dd748d6ef1b020f2c92dfec461d82ad7ffdca07fb89a23b00"} Jul 21 12:31:57.108908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:31:57.108536 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" event={"ID":"2aef0038-decc-47ea-b002-b871ad11603f","Type":"ContainerStarted","Data":"5d7872f327b988be65a4a8b9694c26acf6f7ba26e816697816759a3f9b07c055"} Jul 21 12:32:00.124851 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:00.124816 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-g9l2p" event={"ID":"52e48d1e-6522-49a3-9c4f-655a439d2057","Type":"ContainerStarted","Data":"3e1bd24aa5725782d7d7ba7adc96962fd571314dd0f9726c4464ae50df2c6190"} Jul 21 12:32:00.125300 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:00.124878 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-g9l2p" Jul 21 12:32:00.126615 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:00.126592 2574 generic.go:356] "Generic (PLEG): container finished" podID="2aef0038-decc-47ea-b002-b871ad11603f" containerID="60eb014a842bf46a6c95c282c960b704d516cc341f486e23dcb9212f9c6bf3df" exitCode=0 Jul 21 12:32:00.126725 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:00.126655 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" event={"ID":"2aef0038-decc-47ea-b002-b871ad11603f","Type":"ContainerDied","Data":"60eb014a842bf46a6c95c282c960b704d516cc341f486e23dcb9212f9c6bf3df"} Jul 21 12:32:00.142886 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:00.142829 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-g9l2p" podStartSLOduration=1.650217525 podStartE2EDuration="5.142807229s" totalImagesPullingTime="3.492589704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:31:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:32:00.141905826 +0000 UTC m=+266.792319188" watchObservedRunningTime="2026-07-21 12:32:00.142807229 +0000 UTC m=+266.793220591" Jul 21 12:32:01.133060 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:01.133023 2574 generic.go:356] "Generic (PLEG): container finished" podID="2aef0038-decc-47ea-b002-b871ad11603f" containerID="fb77048261822ee20d9be5d8bcbf6a59c6cd081fd3ec352ecf2deb0cd36821ff" exitCode=0 Jul 21 12:32:01.133548 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:01.133114 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" event={"ID":"2aef0038-decc-47ea-b002-b871ad11603f","Type":"ContainerDied","Data":"fb77048261822ee20d9be5d8bcbf6a59c6cd081fd3ec352ecf2deb0cd36821ff"} Jul 21 12:32:02.285837 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:02.285804 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" Jul 21 12:32:02.405750 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:02.405652 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6prs\" (UniqueName: \"kubernetes.io/projected/2aef0038-decc-47ea-b002-b871ad11603f-kube-api-access-v6prs\") pod \"2aef0038-decc-47ea-b002-b871ad11603f\" (UID: \"2aef0038-decc-47ea-b002-b871ad11603f\") " Jul 21 12:32:02.405920 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:02.405755 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aef0038-decc-47ea-b002-b871ad11603f-bundle\") pod \"2aef0038-decc-47ea-b002-b871ad11603f\" (UID: \"2aef0038-decc-47ea-b002-b871ad11603f\") " Jul 21 12:32:02.405920 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:02.405783 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2aef0038-decc-47ea-b002-b871ad11603f-util\") pod \"2aef0038-decc-47ea-b002-b871ad11603f\" (UID: \"2aef0038-decc-47ea-b002-b871ad11603f\") " Jul 21 12:32:02.406183 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:02.406154 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aef0038-decc-47ea-b002-b871ad11603f-bundle" (OuterVolumeSpecName: "bundle") pod "2aef0038-decc-47ea-b002-b871ad11603f" (UID: "2aef0038-decc-47ea-b002-b871ad11603f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:32:02.407903 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:02.407879 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aef0038-decc-47ea-b002-b871ad11603f-kube-api-access-v6prs" (OuterVolumeSpecName: "kube-api-access-v6prs") pod "2aef0038-decc-47ea-b002-b871ad11603f" (UID: "2aef0038-decc-47ea-b002-b871ad11603f"). InnerVolumeSpecName "kube-api-access-v6prs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:32:02.411634 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:02.411595 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aef0038-decc-47ea-b002-b871ad11603f-util" (OuterVolumeSpecName: "util") pod "2aef0038-decc-47ea-b002-b871ad11603f" (UID: "2aef0038-decc-47ea-b002-b871ad11603f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:32:02.506475 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:02.506428 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aef0038-decc-47ea-b002-b871ad11603f-bundle\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:32:02.506475 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:02.506466 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2aef0038-decc-47ea-b002-b871ad11603f-util\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:32:02.506475 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:02.506478 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6prs\" (UniqueName: \"kubernetes.io/projected/2aef0038-decc-47ea-b002-b871ad11603f-kube-api-access-v6prs\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:32:03.144272 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:03.144230 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" event={"ID":"2aef0038-decc-47ea-b002-b871ad11603f","Type":"ContainerDied","Data":"5d7872f327b988be65a4a8b9694c26acf6f7ba26e816697816759a3f9b07c055"} Jul 21 12:32:03.144272 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:03.144281 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d7872f327b988be65a4a8b9694c26acf6f7ba26e816697816759a3f9b07c055" Jul 21 12:32:03.144523 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:03.144283 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2c8dh" Jul 21 12:32:06.135657 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:06.135630 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-g9l2p" Jul 21 12:32:15.235903 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.235865 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld"] Jul 21 12:32:15.236300 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.236207 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aef0038-decc-47ea-b002-b871ad11603f" containerName="extract" Jul 21 12:32:15.236300 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.236220 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aef0038-decc-47ea-b002-b871ad11603f" containerName="extract" Jul 21 12:32:15.236300 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.236233 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aef0038-decc-47ea-b002-b871ad11603f" containerName="util" Jul 21 12:32:15.236300 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.236238 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aef0038-decc-47ea-b002-b871ad11603f" containerName="util" Jul 21 12:32:15.236300 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.236250 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aef0038-decc-47ea-b002-b871ad11603f" containerName="pull" Jul 21 12:32:15.236300 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.236258 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aef0038-decc-47ea-b002-b871ad11603f" containerName="pull" Jul 21 12:32:15.236481 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.236320 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aef0038-decc-47ea-b002-b871ad11603f" containerName="extract" Jul 21 12:32:15.238618 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.238602 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" Jul 21 12:32:15.241482 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.241454 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 12:32:15.242775 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.242756 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 12:32:15.242912 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.242785 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wpjd9\"" Jul 21 12:32:15.305873 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.305835 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld"] Jul 21 12:32:15.414577 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.414520 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c55ad9b-d048-4177-9715-a091241115ab-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld\" (UID: \"8c55ad9b-d048-4177-9715-a091241115ab\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" Jul 21 12:32:15.414776 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.414598 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c55ad9b-d048-4177-9715-a091241115ab-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld\" (UID: \"8c55ad9b-d048-4177-9715-a091241115ab\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" Jul 21 12:32:15.414776 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.414642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxksq\" (UniqueName: \"kubernetes.io/projected/8c55ad9b-d048-4177-9715-a091241115ab-kube-api-access-vxksq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld\" (UID: \"8c55ad9b-d048-4177-9715-a091241115ab\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" Jul 21 12:32:15.515914 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.515876 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c55ad9b-d048-4177-9715-a091241115ab-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld\" (UID: \"8c55ad9b-d048-4177-9715-a091241115ab\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" Jul 21 12:32:15.516123 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.515932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c55ad9b-d048-4177-9715-a091241115ab-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld\" (UID: \"8c55ad9b-d048-4177-9715-a091241115ab\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" Jul 21 12:32:15.516123 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.515993 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxksq\" (UniqueName: \"kubernetes.io/projected/8c55ad9b-d048-4177-9715-a091241115ab-kube-api-access-vxksq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld\" (UID: \"8c55ad9b-d048-4177-9715-a091241115ab\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" Jul 21 12:32:15.516342 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.516320 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c55ad9b-d048-4177-9715-a091241115ab-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld\" (UID: \"8c55ad9b-d048-4177-9715-a091241115ab\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" Jul 21 12:32:15.516412 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.516354 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c55ad9b-d048-4177-9715-a091241115ab-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld\" (UID: \"8c55ad9b-d048-4177-9715-a091241115ab\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" Jul 21 12:32:15.526247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.526214 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxksq\" (UniqueName: \"kubernetes.io/projected/8c55ad9b-d048-4177-9715-a091241115ab-kube-api-access-vxksq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld\" (UID: \"8c55ad9b-d048-4177-9715-a091241115ab\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" Jul 21 12:32:15.548222 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.548190 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" Jul 21 12:32:15.685639 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:15.685610 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld"] Jul 21 12:32:15.688195 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:32:15.688163 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c55ad9b_d048_4177_9715_a091241115ab.slice/crio-2f2483621dc4046caa920949e81d6c0fba1f4151e5c42766a1667893c04914ea WatchSource:0}: Error finding container 2f2483621dc4046caa920949e81d6c0fba1f4151e5c42766a1667893c04914ea: Status 404 returned error can't find the container with id 2f2483621dc4046caa920949e81d6c0fba1f4151e5c42766a1667893c04914ea Jul 21 12:32:16.199570 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:16.199476 2574 generic.go:356] "Generic (PLEG): container finished" podID="8c55ad9b-d048-4177-9715-a091241115ab" containerID="6b300c02cc97b09275b8a39d0b5043b741c19b7a85224cd758038f5db8b4fb7f" exitCode=0 Jul 21 12:32:16.199722 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:16.199566 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" event={"ID":"8c55ad9b-d048-4177-9715-a091241115ab","Type":"ContainerDied","Data":"6b300c02cc97b09275b8a39d0b5043b741c19b7a85224cd758038f5db8b4fb7f"} Jul 21 12:32:16.199722 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:16.199627 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" event={"ID":"8c55ad9b-d048-4177-9715-a091241115ab","Type":"ContainerStarted","Data":"2f2483621dc4046caa920949e81d6c0fba1f4151e5c42766a1667893c04914ea"} Jul 21 12:32:18.210045 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:18.210010 2574 generic.go:356] "Generic (PLEG): container finished" podID="8c55ad9b-d048-4177-9715-a091241115ab" containerID="0a8780ca669bb1dac90d59025bb34c9314fe9d9c988b8926783e2e3cc9068688" exitCode=0 Jul 21 12:32:18.210460 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:18.210097 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" event={"ID":"8c55ad9b-d048-4177-9715-a091241115ab","Type":"ContainerDied","Data":"0a8780ca669bb1dac90d59025bb34c9314fe9d9c988b8926783e2e3cc9068688"} Jul 21 12:32:19.216529 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:19.216492 2574 generic.go:356] "Generic (PLEG): container finished" podID="8c55ad9b-d048-4177-9715-a091241115ab" containerID="76e6849a6b98d237e4169c7c42f16c80db78fcd4da38e393a942f802311718b2" exitCode=0 Jul 21 12:32:19.216918 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:19.216573 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" event={"ID":"8c55ad9b-d048-4177-9715-a091241115ab","Type":"ContainerDied","Data":"76e6849a6b98d237e4169c7c42f16c80db78fcd4da38e393a942f802311718b2"} Jul 21 12:32:20.360051 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:20.360025 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" Jul 21 12:32:20.456626 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:20.456586 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c55ad9b-d048-4177-9715-a091241115ab-bundle\") pod \"8c55ad9b-d048-4177-9715-a091241115ab\" (UID: \"8c55ad9b-d048-4177-9715-a091241115ab\") " Jul 21 12:32:20.456822 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:20.456684 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxksq\" (UniqueName: \"kubernetes.io/projected/8c55ad9b-d048-4177-9715-a091241115ab-kube-api-access-vxksq\") pod \"8c55ad9b-d048-4177-9715-a091241115ab\" (UID: \"8c55ad9b-d048-4177-9715-a091241115ab\") " Jul 21 12:32:20.456822 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:20.456781 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c55ad9b-d048-4177-9715-a091241115ab-util\") pod \"8c55ad9b-d048-4177-9715-a091241115ab\" (UID: \"8c55ad9b-d048-4177-9715-a091241115ab\") " Jul 21 12:32:20.457613 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:20.457580 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c55ad9b-d048-4177-9715-a091241115ab-bundle" (OuterVolumeSpecName: "bundle") pod "8c55ad9b-d048-4177-9715-a091241115ab" (UID: "8c55ad9b-d048-4177-9715-a091241115ab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:32:20.458951 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:20.458905 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c55ad9b-d048-4177-9715-a091241115ab-kube-api-access-vxksq" (OuterVolumeSpecName: "kube-api-access-vxksq") pod "8c55ad9b-d048-4177-9715-a091241115ab" (UID: "8c55ad9b-d048-4177-9715-a091241115ab"). InnerVolumeSpecName "kube-api-access-vxksq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:32:20.462770 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:20.462744 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c55ad9b-d048-4177-9715-a091241115ab-util" (OuterVolumeSpecName: "util") pod "8c55ad9b-d048-4177-9715-a091241115ab" (UID: "8c55ad9b-d048-4177-9715-a091241115ab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:32:20.557651 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:20.557621 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c55ad9b-d048-4177-9715-a091241115ab-bundle\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:32:20.557651 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:20.557650 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxksq\" (UniqueName: \"kubernetes.io/projected/8c55ad9b-d048-4177-9715-a091241115ab-kube-api-access-vxksq\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:32:20.557854 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:20.557662 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c55ad9b-d048-4177-9715-a091241115ab-util\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:32:21.226580 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:21.226544 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" event={"ID":"8c55ad9b-d048-4177-9715-a091241115ab","Type":"ContainerDied","Data":"2f2483621dc4046caa920949e81d6c0fba1f4151e5c42766a1667893c04914ea"} Jul 21 12:32:21.226748 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:21.226592 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f2483621dc4046caa920949e81d6c0fba1f4151e5c42766a1667893c04914ea" Jul 21 12:32:21.226748 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:21.226593 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4cld" Jul 21 12:32:28.541695 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.541657 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9"] Jul 21 12:32:28.542101 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.542007 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c55ad9b-d048-4177-9715-a091241115ab" containerName="pull" Jul 21 12:32:28.542101 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.542019 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c55ad9b-d048-4177-9715-a091241115ab" containerName="pull" Jul 21 12:32:28.542101 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.542034 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c55ad9b-d048-4177-9715-a091241115ab" containerName="extract" Jul 21 12:32:28.542101 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.542040 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c55ad9b-d048-4177-9715-a091241115ab" containerName="extract" Jul 21 12:32:28.542101 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.542053 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c55ad9b-d048-4177-9715-a091241115ab" containerName="util" Jul 21 12:32:28.542101 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.542058 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c55ad9b-d048-4177-9715-a091241115ab" containerName="util" Jul 21 12:32:28.542280 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.542113 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c55ad9b-d048-4177-9715-a091241115ab" containerName="extract" Jul 21 12:32:28.545779 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.545759 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.549776 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.549752 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 12:32:28.549903 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.549756 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-49kkt\"" Jul 21 12:32:28.550929 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.550907 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 12:32:28.551523 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.551505 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 12:32:28.564250 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.564216 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 12:32:28.574552 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.574521 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9"] Jul 21 12:32:28.619113 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.619076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/69ca45fa-f0f8-4f04-acbe-580377f55252-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.619267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.619133 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/69ca45fa-f0f8-4f04-acbe-580377f55252-manifests\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.619267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.619185 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69ca45fa-f0f8-4f04-acbe-580377f55252-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.619267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.619214 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69ca45fa-f0f8-4f04-acbe-580377f55252-tmp\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.619267 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.619245 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sks4n\" (UniqueName: \"kubernetes.io/projected/69ca45fa-f0f8-4f04-acbe-580377f55252-kube-api-access-sks4n\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.720149 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.720106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69ca45fa-f0f8-4f04-acbe-580377f55252-tmp\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.720327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.720161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sks4n\" (UniqueName: \"kubernetes.io/projected/69ca45fa-f0f8-4f04-acbe-580377f55252-kube-api-access-sks4n\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.720327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.720210 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/69ca45fa-f0f8-4f04-acbe-580377f55252-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.720327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.720250 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/69ca45fa-f0f8-4f04-acbe-580377f55252-manifests\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.720327 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.720289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69ca45fa-f0f8-4f04-acbe-580377f55252-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.720633 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.720611 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/69ca45fa-f0f8-4f04-acbe-580377f55252-manifests\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.722403 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.722373 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69ca45fa-f0f8-4f04-acbe-580377f55252-tmp\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.722662 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.722642 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/69ca45fa-f0f8-4f04-acbe-580377f55252-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.722763 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.722742 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69ca45fa-f0f8-4f04-acbe-580377f55252-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.730200 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.730174 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sks4n\" (UniqueName: \"kubernetes.io/projected/69ca45fa-f0f8-4f04-acbe-580377f55252-kube-api-access-sks4n\") pod \"opendatahub-operator-controller-manager-7cb6668c8b-tcfj9\" (UID: \"69ca45fa-f0f8-4f04-acbe-580377f55252\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:28.856066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:28.855972 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:29.011518 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:29.011488 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9"] Jul 21 12:32:29.014063 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:32:29.014029 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ca45fa_f0f8_4f04_acbe_580377f55252.slice/crio-cb0a501e35d1e1c0f4b78cc9fb7a6da98f371fe766397de490689fbd9fe55855 WatchSource:0}: Error finding container cb0a501e35d1e1c0f4b78cc9fb7a6da98f371fe766397de490689fbd9fe55855: Status 404 returned error can't find the container with id cb0a501e35d1e1c0f4b78cc9fb7a6da98f371fe766397de490689fbd9fe55855 Jul 21 12:32:29.263527 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:29.263492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" event={"ID":"69ca45fa-f0f8-4f04-acbe-580377f55252","Type":"ContainerStarted","Data":"cb0a501e35d1e1c0f4b78cc9fb7a6da98f371fe766397de490689fbd9fe55855"} Jul 21 12:32:32.281987 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:32.281950 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" event={"ID":"69ca45fa-f0f8-4f04-acbe-580377f55252","Type":"ContainerStarted","Data":"a33f5cfd6f6270362120c862ef3c13b8a2dcac33cb4e146ddbdfefa609162cef"} Jul 21 12:32:33.287961 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:33.287905 2574 generic.go:356] "Generic (PLEG): container finished" podID="69ca45fa-f0f8-4f04-acbe-580377f55252" containerID="a33f5cfd6f6270362120c862ef3c13b8a2dcac33cb4e146ddbdfefa609162cef" exitCode=0 Jul 21 12:32:33.288351 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:33.287980 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" event={"ID":"69ca45fa-f0f8-4f04-acbe-580377f55252","Type":"ContainerDied","Data":"a33f5cfd6f6270362120c862ef3c13b8a2dcac33cb4e146ddbdfefa609162cef"} Jul 21 12:32:33.794182 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:33.794154 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:32:33.794723 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:33.794699 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:32:33.800528 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:33.800507 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:32:33.801312 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:33.801289 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:32:33.806346 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:33.806324 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 12:32:35.298551 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:35.298506 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" event={"ID":"69ca45fa-f0f8-4f04-acbe-580377f55252","Type":"ContainerStarted","Data":"8beb70786bfe20da96acc914fe79824d5150d19a1c1d901eae52d595cb538d88"} Jul 21 12:32:35.298992 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:35.298580 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:35.322621 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:35.322571 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" podStartSLOduration=2.033980248 podStartE2EDuration="7.322554344s" totalImagesPullingTime="4.236922008s" totalInitContainerRuntime="1.051652088s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:32:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:32:35.320218742 +0000 UTC m=+301.970632146" watchObservedRunningTime="2026-07-21 12:32:35.322554344 +0000 UTC m=+301.972967704" Jul 21 12:32:45.977541 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:45.977503 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds"] Jul 21 12:32:45.983065 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:45.983042 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:45.986851 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:45.986826 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 21 12:32:45.987792 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:45.987772 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 21 12:32:45.987861 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:45.987844 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 21 12:32:45.987924 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:45.987909 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vmscw\"" Jul 21 12:32:45.989312 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:45.989294 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 21 12:32:45.990270 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:45.990066 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 21 12:32:46.036473 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.036439 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds"] Jul 21 12:32:46.071838 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.071792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khk9p\" (UniqueName: \"kubernetes.io/projected/85f66440-b3c8-4597-b9ef-0efa9acaaeec-kube-api-access-khk9p\") pod \"lws-controller-manager-7c45b47d79-grmds\" (UID: \"85f66440-b3c8-4597-b9ef-0efa9acaaeec\") " pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:46.071838 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.071844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85f66440-b3c8-4597-b9ef-0efa9acaaeec-cert\") pod \"lws-controller-manager-7c45b47d79-grmds\" (UID: \"85f66440-b3c8-4597-b9ef-0efa9acaaeec\") " pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:46.072078 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.071884 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85f66440-b3c8-4597-b9ef-0efa9acaaeec-metrics-cert\") pod \"lws-controller-manager-7c45b47d79-grmds\" (UID: \"85f66440-b3c8-4597-b9ef-0efa9acaaeec\") " pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:46.072078 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.072041 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/85f66440-b3c8-4597-b9ef-0efa9acaaeec-manager-config\") pod \"lws-controller-manager-7c45b47d79-grmds\" (UID: \"85f66440-b3c8-4597-b9ef-0efa9acaaeec\") " pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:46.173196 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.173161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/85f66440-b3c8-4597-b9ef-0efa9acaaeec-manager-config\") pod \"lws-controller-manager-7c45b47d79-grmds\" (UID: \"85f66440-b3c8-4597-b9ef-0efa9acaaeec\") " pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:46.173383 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.173203 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khk9p\" (UniqueName: \"kubernetes.io/projected/85f66440-b3c8-4597-b9ef-0efa9acaaeec-kube-api-access-khk9p\") pod \"lws-controller-manager-7c45b47d79-grmds\" (UID: \"85f66440-b3c8-4597-b9ef-0efa9acaaeec\") " pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:46.173383 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.173230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85f66440-b3c8-4597-b9ef-0efa9acaaeec-cert\") pod \"lws-controller-manager-7c45b47d79-grmds\" (UID: \"85f66440-b3c8-4597-b9ef-0efa9acaaeec\") " pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:46.173383 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.173283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85f66440-b3c8-4597-b9ef-0efa9acaaeec-metrics-cert\") pod \"lws-controller-manager-7c45b47d79-grmds\" (UID: \"85f66440-b3c8-4597-b9ef-0efa9acaaeec\") " pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:46.174011 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.173979 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/85f66440-b3c8-4597-b9ef-0efa9acaaeec-manager-config\") pod \"lws-controller-manager-7c45b47d79-grmds\" (UID: \"85f66440-b3c8-4597-b9ef-0efa9acaaeec\") " pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:46.175895 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.175871 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85f66440-b3c8-4597-b9ef-0efa9acaaeec-cert\") pod \"lws-controller-manager-7c45b47d79-grmds\" (UID: \"85f66440-b3c8-4597-b9ef-0efa9acaaeec\") " pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:46.176010 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.175878 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85f66440-b3c8-4597-b9ef-0efa9acaaeec-metrics-cert\") pod \"lws-controller-manager-7c45b47d79-grmds\" (UID: \"85f66440-b3c8-4597-b9ef-0efa9acaaeec\") " pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:46.183343 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.183312 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khk9p\" (UniqueName: \"kubernetes.io/projected/85f66440-b3c8-4597-b9ef-0efa9acaaeec-kube-api-access-khk9p\") pod \"lws-controller-manager-7c45b47d79-grmds\" (UID: \"85f66440-b3c8-4597-b9ef-0efa9acaaeec\") " pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:46.293102 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.293068 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:46.306019 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.305991 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7cb6668c8b-tcfj9" Jul 21 12:32:46.439033 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.439007 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds"] Jul 21 12:32:46.441221 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:32:46.441194 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f66440_b3c8_4597_b9ef_0efa9acaaeec.slice/crio-0a55b3572e06ffc881a42b24baf7b7183436273325d44f0c6ba3b7f776963d1c WatchSource:0}: Error finding container 0a55b3572e06ffc881a42b24baf7b7183436273325d44f0c6ba3b7f776963d1c: Status 404 returned error can't find the container with id 0a55b3572e06ffc881a42b24baf7b7183436273325d44f0c6ba3b7f776963d1c Jul 21 12:32:46.443243 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:46.443229 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 12:32:47.361512 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:47.361478 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" event={"ID":"85f66440-b3c8-4597-b9ef-0efa9acaaeec","Type":"ContainerStarted","Data":"0a55b3572e06ffc881a42b24baf7b7183436273325d44f0c6ba3b7f776963d1c"} Jul 21 12:32:48.368280 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:48.368246 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" event={"ID":"85f66440-b3c8-4597-b9ef-0efa9acaaeec","Type":"ContainerStarted","Data":"6db957b121653848a288e4d2e5bfdf3553b81ee6d603a8e6060cb1ed3ba6983c"} Jul 21 12:32:48.368695 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:48.368394 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:32:48.436746 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:48.436637 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" podStartSLOduration=1.740634634 podStartE2EDuration="3.436618699s" totalImagesPullingTime="1.695984065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:32:48.430597041 +0000 UTC m=+315.081010403" watchObservedRunningTime="2026-07-21 12:32:48.436618699 +0000 UTC m=+315.087032060" Jul 21 12:32:57.299756 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.299713 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4"] Jul 21 12:32:57.302158 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.302133 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.307074 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.307053 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 21 12:32:57.307424 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.307407 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 21 12:32:57.307629 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.307600 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 21 12:32:57.307994 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.307977 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 12:32:57.308047 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.307976 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fn49f\"" Jul 21 12:32:57.308141 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.308125 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 12:32:57.308231 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.308216 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 12:32:57.323297 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.323262 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4"] Jul 21 12:32:57.371408 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.371364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2e72ec98-c294-4082-b7f5-35226069ed1a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.371589 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.371421 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e72ec98-c294-4082-b7f5-35226069ed1a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.371589 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.371448 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2e72ec98-c294-4082-b7f5-35226069ed1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.371589 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.371476 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e72ec98-c294-4082-b7f5-35226069ed1a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.371589 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.371534 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e72ec98-c294-4082-b7f5-35226069ed1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.371589 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.371557 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht9qm\" (UniqueName: \"kubernetes.io/projected/2e72ec98-c294-4082-b7f5-35226069ed1a-kube-api-access-ht9qm\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.371589 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.371588 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2e72ec98-c294-4082-b7f5-35226069ed1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.472967 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.472908 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2e72ec98-c294-4082-b7f5-35226069ed1a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.473139 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.472974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e72ec98-c294-4082-b7f5-35226069ed1a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.473139 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.472995 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2e72ec98-c294-4082-b7f5-35226069ed1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.473139 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.473022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e72ec98-c294-4082-b7f5-35226069ed1a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.473139 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.473060 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e72ec98-c294-4082-b7f5-35226069ed1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.473139 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.473081 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht9qm\" (UniqueName: \"kubernetes.io/projected/2e72ec98-c294-4082-b7f5-35226069ed1a-kube-api-access-ht9qm\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.473139 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.473124 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2e72ec98-c294-4082-b7f5-35226069ed1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.473888 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.473862 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2e72ec98-c294-4082-b7f5-35226069ed1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.475503 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.475475 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e72ec98-c294-4082-b7f5-35226069ed1a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.475596 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.475475 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2e72ec98-c294-4082-b7f5-35226069ed1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.475596 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.475568 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e72ec98-c294-4082-b7f5-35226069ed1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.475674 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.475635 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2e72ec98-c294-4082-b7f5-35226069ed1a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.482755 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.482721 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e72ec98-c294-4082-b7f5-35226069ed1a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.487270 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.487241 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht9qm\" (UniqueName: \"kubernetes.io/projected/2e72ec98-c294-4082-b7f5-35226069ed1a-kube-api-access-ht9qm\") pod \"istiod-openshift-gateway-94bb8fbfd-vnng4\" (UID: \"2e72ec98-c294-4082-b7f5-35226069ed1a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.611831 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.611745 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:32:57.771792 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:57.771761 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4"] Jul 21 12:32:57.774653 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:32:57.774624 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e72ec98_c294_4082_b7f5_35226069ed1a.slice/crio-a87766d29b88e400ab19e83c564320f35f0ddc7e9cea9cf499272f7fbe90b09d WatchSource:0}: Error finding container a87766d29b88e400ab19e83c564320f35f0ddc7e9cea9cf499272f7fbe90b09d: Status 404 returned error can't find the container with id a87766d29b88e400ab19e83c564320f35f0ddc7e9cea9cf499272f7fbe90b09d Jul 21 12:32:58.416800 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:58.416767 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" event={"ID":"2e72ec98-c294-4082-b7f5-35226069ed1a","Type":"ContainerStarted","Data":"a87766d29b88e400ab19e83c564320f35f0ddc7e9cea9cf499272f7fbe90b09d"} Jul 21 12:32:59.375963 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:32:59.375913 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c45b47d79-grmds" Jul 21 12:33:00.130959 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:33:00.130893 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 12:33:00.131331 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:33:00.130993 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 12:33:00.430884 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:33:00.430796 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" event={"ID":"2e72ec98-c294-4082-b7f5-35226069ed1a","Type":"ContainerStarted","Data":"1bcf5216329cff1d9886b234dda86ce9e7417fc4f2f34d1028f6547d8fbe83c3"} Jul 21 12:33:00.431069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:33:00.431002 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:33:00.474033 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:33:00.473930 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" podStartSLOduration=1.119817014 podStartE2EDuration="3.473911055s" totalImagesPullingTime="2.354094041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:33:00.472479568 +0000 UTC m=+327.122892930" watchObservedRunningTime="2026-07-21 12:33:00.473911055 +0000 UTC m=+327.124324416" Jul 21 12:33:01.437279 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:33:01.437241 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-vnng4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 12:33:01.437741 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:33:01.437320 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" podUID="2e72ec98-c294-4082-b7f5-35226069ed1a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 12:33:04.436783 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:33:04.436744 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vnng4" Jul 21 12:34:06.048305 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.048271 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5"] Jul 21 12:34:06.053231 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.053210 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" Jul 21 12:34:06.057886 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.057859 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 12:34:06.058031 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.057911 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ckjqb\"" Jul 21 12:34:06.058206 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.058185 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 12:34:06.069480 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.069448 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5"] Jul 21 12:34:06.173580 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.173535 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54dd00c7-7600-40e0-9053-4f151bea65cd-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5\" (UID: \"54dd00c7-7600-40e0-9053-4f151bea65cd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" Jul 21 12:34:06.173759 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.173657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlg5g\" (UniqueName: \"kubernetes.io/projected/54dd00c7-7600-40e0-9053-4f151bea65cd-kube-api-access-rlg5g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5\" (UID: \"54dd00c7-7600-40e0-9053-4f151bea65cd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" Jul 21 12:34:06.173759 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.173696 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54dd00c7-7600-40e0-9053-4f151bea65cd-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5\" (UID: \"54dd00c7-7600-40e0-9053-4f151bea65cd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" Jul 21 12:34:06.275258 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.275220 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54dd00c7-7600-40e0-9053-4f151bea65cd-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5\" (UID: \"54dd00c7-7600-40e0-9053-4f151bea65cd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" Jul 21 12:34:06.275424 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.275296 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlg5g\" (UniqueName: \"kubernetes.io/projected/54dd00c7-7600-40e0-9053-4f151bea65cd-kube-api-access-rlg5g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5\" (UID: \"54dd00c7-7600-40e0-9053-4f151bea65cd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" Jul 21 12:34:06.275424 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.275324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54dd00c7-7600-40e0-9053-4f151bea65cd-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5\" (UID: \"54dd00c7-7600-40e0-9053-4f151bea65cd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" Jul 21 12:34:06.275715 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.275691 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54dd00c7-7600-40e0-9053-4f151bea65cd-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5\" (UID: \"54dd00c7-7600-40e0-9053-4f151bea65cd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" Jul 21 12:34:06.275755 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.275696 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54dd00c7-7600-40e0-9053-4f151bea65cd-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5\" (UID: \"54dd00c7-7600-40e0-9053-4f151bea65cd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" Jul 21 12:34:06.284078 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.284042 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlg5g\" (UniqueName: \"kubernetes.io/projected/54dd00c7-7600-40e0-9053-4f151bea65cd-kube-api-access-rlg5g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5\" (UID: \"54dd00c7-7600-40e0-9053-4f151bea65cd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" Jul 21 12:34:06.362712 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.362619 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" Jul 21 12:34:06.498967 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.498797 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5"] Jul 21 12:34:06.501703 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:34:06.501675 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54dd00c7_7600_40e0_9053_4f151bea65cd.slice/crio-95ed2d15ba309122c1b952b36818e3790afaaadde2150ca9bbf68607c716b4fd WatchSource:0}: Error finding container 95ed2d15ba309122c1b952b36818e3790afaaadde2150ca9bbf68607c716b4fd: Status 404 returned error can't find the container with id 95ed2d15ba309122c1b952b36818e3790afaaadde2150ca9bbf68607c716b4fd Jul 21 12:34:06.605385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.605355 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4"] Jul 21 12:34:06.609478 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.609457 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" Jul 21 12:34:06.617655 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.617589 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4"] Jul 21 12:34:06.746647 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.746610 2574 generic.go:356] "Generic (PLEG): container finished" podID="54dd00c7-7600-40e0-9053-4f151bea65cd" containerID="7507d3af5a8df89be10cb7bf8dac9d4481489c0004bbd619b44f3dc54711eff5" exitCode=0 Jul 21 12:34:06.746821 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.746696 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" event={"ID":"54dd00c7-7600-40e0-9053-4f151bea65cd","Type":"ContainerDied","Data":"7507d3af5a8df89be10cb7bf8dac9d4481489c0004bbd619b44f3dc54711eff5"} Jul 21 12:34:06.746821 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.746737 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" event={"ID":"54dd00c7-7600-40e0-9053-4f151bea65cd","Type":"ContainerStarted","Data":"95ed2d15ba309122c1b952b36818e3790afaaadde2150ca9bbf68607c716b4fd"} Jul 21 12:34:06.780173 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.780132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89ddd4ed-1f36-4839-a9a2-edb28d549c98-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4\" (UID: \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" Jul 21 12:34:06.780347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.780214 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49gj5\" (UniqueName: \"kubernetes.io/projected/89ddd4ed-1f36-4839-a9a2-edb28d549c98-kube-api-access-49gj5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4\" (UID: \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" Jul 21 12:34:06.780347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.780259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ddd4ed-1f36-4839-a9a2-edb28d549c98-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4\" (UID: \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" Jul 21 12:34:06.881259 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.881170 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gj5\" (UniqueName: \"kubernetes.io/projected/89ddd4ed-1f36-4839-a9a2-edb28d549c98-kube-api-access-49gj5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4\" (UID: \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" Jul 21 12:34:06.881259 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.881222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ddd4ed-1f36-4839-a9a2-edb28d549c98-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4\" (UID: \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" Jul 21 12:34:06.881428 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.881371 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89ddd4ed-1f36-4839-a9a2-edb28d549c98-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4\" (UID: \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" Jul 21 12:34:06.881604 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.881578 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ddd4ed-1f36-4839-a9a2-edb28d549c98-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4\" (UID: \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" Jul 21 12:34:06.881756 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.881737 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89ddd4ed-1f36-4839-a9a2-edb28d549c98-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4\" (UID: \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" Jul 21 12:34:06.890103 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.890071 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gj5\" (UniqueName: \"kubernetes.io/projected/89ddd4ed-1f36-4839-a9a2-edb28d549c98-kube-api-access-49gj5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4\" (UID: \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" Jul 21 12:34:06.961817 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:06.961782 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" Jul 21 12:34:07.006636 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.006604 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5"] Jul 21 12:34:07.010496 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.010470 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" Jul 21 12:34:07.023037 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.022794 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5"] Jul 21 12:34:07.106233 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.106201 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4"] Jul 21 12:34:07.109568 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:34:07.109536 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ddd4ed_1f36_4839_a9a2_edb28d549c98.slice/crio-978f8af16ec5c54f0b012d2153c00d0c0a23965f0f60821a8625903ef88b1350 WatchSource:0}: Error finding container 978f8af16ec5c54f0b012d2153c00d0c0a23965f0f60821a8625903ef88b1350: Status 404 returned error can't find the container with id 978f8af16ec5c54f0b012d2153c00d0c0a23965f0f60821a8625903ef88b1350 Jul 21 12:34:07.184394 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.184351 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76b9de7e-f20d-415d-861d-ff5aff65c788-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5\" (UID: \"76b9de7e-f20d-415d-861d-ff5aff65c788\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" Jul 21 12:34:07.184394 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.184399 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76b9de7e-f20d-415d-861d-ff5aff65c788-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5\" (UID: \"76b9de7e-f20d-415d-861d-ff5aff65c788\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" Jul 21 12:34:07.184594 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.184539 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zct7z\" (UniqueName: \"kubernetes.io/projected/76b9de7e-f20d-415d-861d-ff5aff65c788-kube-api-access-zct7z\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5\" (UID: \"76b9de7e-f20d-415d-861d-ff5aff65c788\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" Jul 21 12:34:07.285443 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.285403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76b9de7e-f20d-415d-861d-ff5aff65c788-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5\" (UID: \"76b9de7e-f20d-415d-861d-ff5aff65c788\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" Jul 21 12:34:07.285616 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.285456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76b9de7e-f20d-415d-861d-ff5aff65c788-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5\" (UID: \"76b9de7e-f20d-415d-861d-ff5aff65c788\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" Jul 21 12:34:07.285616 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.285524 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zct7z\" (UniqueName: \"kubernetes.io/projected/76b9de7e-f20d-415d-861d-ff5aff65c788-kube-api-access-zct7z\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5\" (UID: \"76b9de7e-f20d-415d-861d-ff5aff65c788\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" Jul 21 12:34:07.285882 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.285859 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76b9de7e-f20d-415d-861d-ff5aff65c788-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5\" (UID: \"76b9de7e-f20d-415d-861d-ff5aff65c788\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" Jul 21 12:34:07.285930 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.285873 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76b9de7e-f20d-415d-861d-ff5aff65c788-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5\" (UID: \"76b9de7e-f20d-415d-861d-ff5aff65c788\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" Jul 21 12:34:07.294966 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.294921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zct7z\" (UniqueName: \"kubernetes.io/projected/76b9de7e-f20d-415d-861d-ff5aff65c788-kube-api-access-zct7z\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5\" (UID: \"76b9de7e-f20d-415d-861d-ff5aff65c788\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" Jul 21 12:34:07.326558 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.326523 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" Jul 21 12:34:07.400996 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.400899 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f"] Jul 21 12:34:07.404716 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.404693 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" Jul 21 12:34:07.413389 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.413287 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f"] Jul 21 12:34:07.509410 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.509375 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5"] Jul 21 12:34:07.512441 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:34:07.512415 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b9de7e_f20d_415d_861d_ff5aff65c788.slice/crio-df23e6c5811aa4e7dc5523893ab70bdca690f4e3cbf21602bf63041d50a22555 WatchSource:0}: Error finding container df23e6c5811aa4e7dc5523893ab70bdca690f4e3cbf21602bf63041d50a22555: Status 404 returned error can't find the container with id df23e6c5811aa4e7dc5523893ab70bdca690f4e3cbf21602bf63041d50a22555 Jul 21 12:34:07.587709 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.587686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5644f7de-07a8-4d2b-829b-d52210da47f5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f\" (UID: \"5644f7de-07a8-4d2b-829b-d52210da47f5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" Jul 21 12:34:07.587834 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.587739 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grzxc\" (UniqueName: \"kubernetes.io/projected/5644f7de-07a8-4d2b-829b-d52210da47f5-kube-api-access-grzxc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f\" (UID: \"5644f7de-07a8-4d2b-829b-d52210da47f5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" Jul 21 12:34:07.587834 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.587786 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5644f7de-07a8-4d2b-829b-d52210da47f5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f\" (UID: \"5644f7de-07a8-4d2b-829b-d52210da47f5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" Jul 21 12:34:07.688395 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.688364 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grzxc\" (UniqueName: \"kubernetes.io/projected/5644f7de-07a8-4d2b-829b-d52210da47f5-kube-api-access-grzxc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f\" (UID: \"5644f7de-07a8-4d2b-829b-d52210da47f5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" Jul 21 12:34:07.688539 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.688410 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5644f7de-07a8-4d2b-829b-d52210da47f5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f\" (UID: \"5644f7de-07a8-4d2b-829b-d52210da47f5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" Jul 21 12:34:07.688539 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.688475 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5644f7de-07a8-4d2b-829b-d52210da47f5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f\" (UID: \"5644f7de-07a8-4d2b-829b-d52210da47f5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" Jul 21 12:34:07.688908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.688886 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5644f7de-07a8-4d2b-829b-d52210da47f5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f\" (UID: \"5644f7de-07a8-4d2b-829b-d52210da47f5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" Jul 21 12:34:07.688908 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.688900 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5644f7de-07a8-4d2b-829b-d52210da47f5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f\" (UID: \"5644f7de-07a8-4d2b-829b-d52210da47f5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" Jul 21 12:34:07.697149 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.697123 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grzxc\" (UniqueName: \"kubernetes.io/projected/5644f7de-07a8-4d2b-829b-d52210da47f5-kube-api-access-grzxc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f\" (UID: \"5644f7de-07a8-4d2b-829b-d52210da47f5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" Jul 21 12:34:07.719146 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.719113 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" Jul 21 12:34:07.753439 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.753403 2574 generic.go:356] "Generic (PLEG): container finished" podID="54dd00c7-7600-40e0-9053-4f151bea65cd" containerID="6d3cd3b105d3803731794d3cbeebb2f1bcde4cfaea1a902a708e5ae2f3c302c7" exitCode=0 Jul 21 12:34:07.753604 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.753481 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" event={"ID":"54dd00c7-7600-40e0-9053-4f151bea65cd","Type":"ContainerDied","Data":"6d3cd3b105d3803731794d3cbeebb2f1bcde4cfaea1a902a708e5ae2f3c302c7"} Jul 21 12:34:07.758284 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.758115 2574 generic.go:356] "Generic (PLEG): container finished" podID="89ddd4ed-1f36-4839-a9a2-edb28d549c98" containerID="675add935b079748c70afadcb53fbe2946cb63c59f3fe30f56ab50c29b7c9921" exitCode=0 Jul 21 12:34:07.758284 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.758215 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" event={"ID":"89ddd4ed-1f36-4839-a9a2-edb28d549c98","Type":"ContainerDied","Data":"675add935b079748c70afadcb53fbe2946cb63c59f3fe30f56ab50c29b7c9921"} Jul 21 12:34:07.758284 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.758272 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" event={"ID":"89ddd4ed-1f36-4839-a9a2-edb28d549c98","Type":"ContainerStarted","Data":"978f8af16ec5c54f0b012d2153c00d0c0a23965f0f60821a8625903ef88b1350"} Jul 21 12:34:07.760647 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.760608 2574 generic.go:356] "Generic (PLEG): container finished" podID="76b9de7e-f20d-415d-861d-ff5aff65c788" containerID="a7bf29c019705463980aaff65aa632af5798966e622bfacb846e21ddd01eb35a" exitCode=0 Jul 21 12:34:07.760762 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.760645 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" event={"ID":"76b9de7e-f20d-415d-861d-ff5aff65c788","Type":"ContainerDied","Data":"a7bf29c019705463980aaff65aa632af5798966e622bfacb846e21ddd01eb35a"} Jul 21 12:34:07.760762 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.760684 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" event={"ID":"76b9de7e-f20d-415d-861d-ff5aff65c788","Type":"ContainerStarted","Data":"df23e6c5811aa4e7dc5523893ab70bdca690f4e3cbf21602bf63041d50a22555"} Jul 21 12:34:07.863965 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:07.863914 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f"] Jul 21 12:34:07.865865 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:34:07.865835 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5644f7de_07a8_4d2b_829b_d52210da47f5.slice/crio-a34edd37cfa6e3f58706ebc43ac87f27703c40d0728e01bd2a5039f7c1edabdf WatchSource:0}: Error finding container a34edd37cfa6e3f58706ebc43ac87f27703c40d0728e01bd2a5039f7c1edabdf: Status 404 returned error can't find the container with id a34edd37cfa6e3f58706ebc43ac87f27703c40d0728e01bd2a5039f7c1edabdf Jul 21 12:34:08.766655 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:08.766619 2574 generic.go:356] "Generic (PLEG): container finished" podID="5644f7de-07a8-4d2b-829b-d52210da47f5" containerID="7bac2845e932d094843b33e07996a9fdc86cafca773ae00c27db0fc509a580fd" exitCode=0 Jul 21 12:34:08.766655 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:08.766647 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" event={"ID":"5644f7de-07a8-4d2b-829b-d52210da47f5","Type":"ContainerDied","Data":"7bac2845e932d094843b33e07996a9fdc86cafca773ae00c27db0fc509a580fd"} Jul 21 12:34:08.767197 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:08.766710 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" event={"ID":"5644f7de-07a8-4d2b-829b-d52210da47f5","Type":"ContainerStarted","Data":"a34edd37cfa6e3f58706ebc43ac87f27703c40d0728e01bd2a5039f7c1edabdf"} Jul 21 12:34:08.768648 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:08.768619 2574 generic.go:356] "Generic (PLEG): container finished" podID="89ddd4ed-1f36-4839-a9a2-edb28d549c98" containerID="e89730104c51b92f433a6801bd4b746330fcb7bc63b45830fe7d6349d5901d00" exitCode=0 Jul 21 12:34:08.768800 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:08.768696 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" event={"ID":"89ddd4ed-1f36-4839-a9a2-edb28d549c98","Type":"ContainerDied","Data":"e89730104c51b92f433a6801bd4b746330fcb7bc63b45830fe7d6349d5901d00"} Jul 21 12:34:08.770625 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:08.770600 2574 generic.go:356] "Generic (PLEG): container finished" podID="76b9de7e-f20d-415d-861d-ff5aff65c788" containerID="b9a271fd01cce4da907330f8d5dc6a93890651482fecaa34425f9f1c6e5d7e03" exitCode=0 Jul 21 12:34:08.770732 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:08.770691 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" event={"ID":"76b9de7e-f20d-415d-861d-ff5aff65c788","Type":"ContainerDied","Data":"b9a271fd01cce4da907330f8d5dc6a93890651482fecaa34425f9f1c6e5d7e03"} Jul 21 12:34:08.773010 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:08.772983 2574 generic.go:356] "Generic (PLEG): container finished" podID="54dd00c7-7600-40e0-9053-4f151bea65cd" containerID="26f71716281f3d2478a4367d1a67cf6b537e5e16aa147b0577a9fe1344f66233" exitCode=0 Jul 21 12:34:08.773093 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:08.773033 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" event={"ID":"54dd00c7-7600-40e0-9053-4f151bea65cd","Type":"ContainerDied","Data":"26f71716281f3d2478a4367d1a67cf6b537e5e16aa147b0577a9fe1344f66233"} Jul 21 12:34:09.780706 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:09.780674 2574 generic.go:356] "Generic (PLEG): container finished" podID="89ddd4ed-1f36-4839-a9a2-edb28d549c98" containerID="9129d7e14a354bf5a53c07fb55f6001fe8674184039f0b54f7b96a9d95ecb5a5" exitCode=0 Jul 21 12:34:09.781171 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:09.780768 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" event={"ID":"89ddd4ed-1f36-4839-a9a2-edb28d549c98","Type":"ContainerDied","Data":"9129d7e14a354bf5a53c07fb55f6001fe8674184039f0b54f7b96a9d95ecb5a5"} Jul 21 12:34:09.782913 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:09.782888 2574 generic.go:356] "Generic (PLEG): container finished" podID="76b9de7e-f20d-415d-861d-ff5aff65c788" containerID="11ff9d92d91e7979277be695ff44daed129dad2b75083f08f8160f4e187c9148" exitCode=0 Jul 21 12:34:09.783044 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:09.782969 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" event={"ID":"76b9de7e-f20d-415d-861d-ff5aff65c788","Type":"ContainerDied","Data":"11ff9d92d91e7979277be695ff44daed129dad2b75083f08f8160f4e187c9148"} Jul 21 12:34:09.958500 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:09.958471 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" Jul 21 12:34:10.113165 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.113127 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54dd00c7-7600-40e0-9053-4f151bea65cd-util\") pod \"54dd00c7-7600-40e0-9053-4f151bea65cd\" (UID: \"54dd00c7-7600-40e0-9053-4f151bea65cd\") " Jul 21 12:34:10.113165 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.113171 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54dd00c7-7600-40e0-9053-4f151bea65cd-bundle\") pod \"54dd00c7-7600-40e0-9053-4f151bea65cd\" (UID: \"54dd00c7-7600-40e0-9053-4f151bea65cd\") " Jul 21 12:34:10.113367 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.113245 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlg5g\" (UniqueName: \"kubernetes.io/projected/54dd00c7-7600-40e0-9053-4f151bea65cd-kube-api-access-rlg5g\") pod \"54dd00c7-7600-40e0-9053-4f151bea65cd\" (UID: \"54dd00c7-7600-40e0-9053-4f151bea65cd\") " Jul 21 12:34:10.113748 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.113712 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dd00c7-7600-40e0-9053-4f151bea65cd-bundle" (OuterVolumeSpecName: "bundle") pod "54dd00c7-7600-40e0-9053-4f151bea65cd" (UID: "54dd00c7-7600-40e0-9053-4f151bea65cd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:34:10.115676 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.115645 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54dd00c7-7600-40e0-9053-4f151bea65cd-kube-api-access-rlg5g" (OuterVolumeSpecName: "kube-api-access-rlg5g") pod "54dd00c7-7600-40e0-9053-4f151bea65cd" (UID: "54dd00c7-7600-40e0-9053-4f151bea65cd"). InnerVolumeSpecName "kube-api-access-rlg5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:34:10.118430 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.118404 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dd00c7-7600-40e0-9053-4f151bea65cd-util" (OuterVolumeSpecName: "util") pod "54dd00c7-7600-40e0-9053-4f151bea65cd" (UID: "54dd00c7-7600-40e0-9053-4f151bea65cd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:34:10.214886 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.214837 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54dd00c7-7600-40e0-9053-4f151bea65cd-util\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:34:10.214886 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.214877 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54dd00c7-7600-40e0-9053-4f151bea65cd-bundle\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:34:10.214886 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.214891 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlg5g\" (UniqueName: \"kubernetes.io/projected/54dd00c7-7600-40e0-9053-4f151bea65cd-kube-api-access-rlg5g\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:34:10.790232 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.790198 2574 generic.go:356] "Generic (PLEG): container finished" podID="5644f7de-07a8-4d2b-829b-d52210da47f5" containerID="58b214230c5df732d1e50444e39a43602004d69207e6be3f35a05864cb7ac98d" exitCode=0 Jul 21 12:34:10.790697 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.790282 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" event={"ID":"5644f7de-07a8-4d2b-829b-d52210da47f5","Type":"ContainerDied","Data":"58b214230c5df732d1e50444e39a43602004d69207e6be3f35a05864cb7ac98d"} Jul 21 12:34:10.792191 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.792172 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" Jul 21 12:34:10.792262 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.792192 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5" event={"ID":"54dd00c7-7600-40e0-9053-4f151bea65cd","Type":"ContainerDied","Data":"95ed2d15ba309122c1b952b36818e3790afaaadde2150ca9bbf68607c716b4fd"} Jul 21 12:34:10.792262 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.792234 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95ed2d15ba309122c1b952b36818e3790afaaadde2150ca9bbf68607c716b4fd" Jul 21 12:34:10.936407 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:10.936378 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" Jul 21 12:34:11.024201 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.024174 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" Jul 21 12:34:11.120845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.120760 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ddd4ed-1f36-4839-a9a2-edb28d549c98-bundle\") pod \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\" (UID: \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\") " Jul 21 12:34:11.120845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.120795 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76b9de7e-f20d-415d-861d-ff5aff65c788-util\") pod \"76b9de7e-f20d-415d-861d-ff5aff65c788\" (UID: \"76b9de7e-f20d-415d-861d-ff5aff65c788\") " Jul 21 12:34:11.120845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.120831 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49gj5\" (UniqueName: \"kubernetes.io/projected/89ddd4ed-1f36-4839-a9a2-edb28d549c98-kube-api-access-49gj5\") pod \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\" (UID: \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\") " Jul 21 12:34:11.121135 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.120858 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zct7z\" (UniqueName: \"kubernetes.io/projected/76b9de7e-f20d-415d-861d-ff5aff65c788-kube-api-access-zct7z\") pod \"76b9de7e-f20d-415d-861d-ff5aff65c788\" (UID: \"76b9de7e-f20d-415d-861d-ff5aff65c788\") " Jul 21 12:34:11.121135 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.120879 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76b9de7e-f20d-415d-861d-ff5aff65c788-bundle\") pod \"76b9de7e-f20d-415d-861d-ff5aff65c788\" (UID: \"76b9de7e-f20d-415d-861d-ff5aff65c788\") " Jul 21 12:34:11.121135 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.120897 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89ddd4ed-1f36-4839-a9a2-edb28d549c98-util\") pod \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\" (UID: \"89ddd4ed-1f36-4839-a9a2-edb28d549c98\") " Jul 21 12:34:11.121499 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.121465 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b9de7e-f20d-415d-861d-ff5aff65c788-bundle" (OuterVolumeSpecName: "bundle") pod "76b9de7e-f20d-415d-861d-ff5aff65c788" (UID: "76b9de7e-f20d-415d-861d-ff5aff65c788"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:34:11.121636 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.121602 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ddd4ed-1f36-4839-a9a2-edb28d549c98-bundle" (OuterVolumeSpecName: "bundle") pod "89ddd4ed-1f36-4839-a9a2-edb28d549c98" (UID: "89ddd4ed-1f36-4839-a9a2-edb28d549c98"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:34:11.123678 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.123652 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b9de7e-f20d-415d-861d-ff5aff65c788-kube-api-access-zct7z" (OuterVolumeSpecName: "kube-api-access-zct7z") pod "76b9de7e-f20d-415d-861d-ff5aff65c788" (UID: "76b9de7e-f20d-415d-861d-ff5aff65c788"). InnerVolumeSpecName "kube-api-access-zct7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:34:11.123678 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.123667 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ddd4ed-1f36-4839-a9a2-edb28d549c98-kube-api-access-49gj5" (OuterVolumeSpecName: "kube-api-access-49gj5") pod "89ddd4ed-1f36-4839-a9a2-edb28d549c98" (UID: "89ddd4ed-1f36-4839-a9a2-edb28d549c98"). InnerVolumeSpecName "kube-api-access-49gj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:34:11.126950 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.126914 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b9de7e-f20d-415d-861d-ff5aff65c788-util" (OuterVolumeSpecName: "util") pod "76b9de7e-f20d-415d-861d-ff5aff65c788" (UID: "76b9de7e-f20d-415d-861d-ff5aff65c788"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:34:11.127184 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.127158 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ddd4ed-1f36-4839-a9a2-edb28d549c98-util" (OuterVolumeSpecName: "util") pod "89ddd4ed-1f36-4839-a9a2-edb28d549c98" (UID: "89ddd4ed-1f36-4839-a9a2-edb28d549c98"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:34:11.221707 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.221664 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89ddd4ed-1f36-4839-a9a2-edb28d549c98-bundle\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:34:11.221707 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.221694 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76b9de7e-f20d-415d-861d-ff5aff65c788-util\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:34:11.221707 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.221704 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49gj5\" (UniqueName: \"kubernetes.io/projected/89ddd4ed-1f36-4839-a9a2-edb28d549c98-kube-api-access-49gj5\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:34:11.221707 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.221714 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zct7z\" (UniqueName: \"kubernetes.io/projected/76b9de7e-f20d-415d-861d-ff5aff65c788-kube-api-access-zct7z\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:34:11.222004 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.221724 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76b9de7e-f20d-415d-861d-ff5aff65c788-bundle\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:34:11.222004 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.221732 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89ddd4ed-1f36-4839-a9a2-edb28d549c98-util\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:34:11.799111 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.799073 2574 generic.go:356] "Generic (PLEG): container finished" podID="5644f7de-07a8-4d2b-829b-d52210da47f5" containerID="bfd02838fd65abc81f055ae765081b318b2f52ccfb04fb2febe257a6bd689f2c" exitCode=0 Jul 21 12:34:11.799531 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.799163 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" event={"ID":"5644f7de-07a8-4d2b-829b-d52210da47f5","Type":"ContainerDied","Data":"bfd02838fd65abc81f055ae765081b318b2f52ccfb04fb2febe257a6bd689f2c"} Jul 21 12:34:11.800988 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.800968 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" Jul 21 12:34:11.801068 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.800985 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4" event={"ID":"89ddd4ed-1f36-4839-a9a2-edb28d549c98","Type":"ContainerDied","Data":"978f8af16ec5c54f0b012d2153c00d0c0a23965f0f60821a8625903ef88b1350"} Jul 21 12:34:11.801068 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.801026 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="978f8af16ec5c54f0b012d2153c00d0c0a23965f0f60821a8625903ef88b1350" Jul 21 12:34:11.802810 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.802788 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" event={"ID":"76b9de7e-f20d-415d-861d-ff5aff65c788","Type":"ContainerDied","Data":"df23e6c5811aa4e7dc5523893ab70bdca690f4e3cbf21602bf63041d50a22555"} Jul 21 12:34:11.802903 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.802825 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df23e6c5811aa4e7dc5523893ab70bdca690f4e3cbf21602bf63041d50a22555" Jul 21 12:34:11.802903 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:11.802832 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5" Jul 21 12:34:12.943822 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:12.943796 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" Jul 21 12:34:13.036128 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:13.036096 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5644f7de-07a8-4d2b-829b-d52210da47f5-util\") pod \"5644f7de-07a8-4d2b-829b-d52210da47f5\" (UID: \"5644f7de-07a8-4d2b-829b-d52210da47f5\") " Jul 21 12:34:13.036335 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:13.036169 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grzxc\" (UniqueName: \"kubernetes.io/projected/5644f7de-07a8-4d2b-829b-d52210da47f5-kube-api-access-grzxc\") pod \"5644f7de-07a8-4d2b-829b-d52210da47f5\" (UID: \"5644f7de-07a8-4d2b-829b-d52210da47f5\") " Jul 21 12:34:13.038281 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:13.038248 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5644f7de-07a8-4d2b-829b-d52210da47f5-kube-api-access-grzxc" (OuterVolumeSpecName: "kube-api-access-grzxc") pod "5644f7de-07a8-4d2b-829b-d52210da47f5" (UID: "5644f7de-07a8-4d2b-829b-d52210da47f5"). InnerVolumeSpecName "kube-api-access-grzxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:34:13.041535 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:13.041510 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5644f7de-07a8-4d2b-829b-d52210da47f5-util" (OuterVolumeSpecName: "util") pod "5644f7de-07a8-4d2b-829b-d52210da47f5" (UID: "5644f7de-07a8-4d2b-829b-d52210da47f5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:34:13.136825 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:13.136730 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5644f7de-07a8-4d2b-829b-d52210da47f5-bundle\") pod \"5644f7de-07a8-4d2b-829b-d52210da47f5\" (UID: \"5644f7de-07a8-4d2b-829b-d52210da47f5\") " Jul 21 12:34:13.137007 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:13.136921 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grzxc\" (UniqueName: \"kubernetes.io/projected/5644f7de-07a8-4d2b-829b-d52210da47f5-kube-api-access-grzxc\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:34:13.137007 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:13.136968 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5644f7de-07a8-4d2b-829b-d52210da47f5-util\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:34:13.137236 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:13.137211 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5644f7de-07a8-4d2b-829b-d52210da47f5-bundle" (OuterVolumeSpecName: "bundle") pod "5644f7de-07a8-4d2b-829b-d52210da47f5" (UID: "5644f7de-07a8-4d2b-829b-d52210da47f5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:34:13.238070 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:13.238025 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5644f7de-07a8-4d2b-829b-d52210da47f5-bundle\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:34:13.814729 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:13.814695 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" Jul 21 12:34:13.814915 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:13.814723 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f" event={"ID":"5644f7de-07a8-4d2b-829b-d52210da47f5","Type":"ContainerDied","Data":"a34edd37cfa6e3f58706ebc43ac87f27703c40d0728e01bd2a5039f7c1edabdf"} Jul 21 12:34:13.814915 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:13.814806 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a34edd37cfa6e3f58706ebc43ac87f27703c40d0728e01bd2a5039f7c1edabdf" Jul 21 12:34:46.328813 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.328776 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb"] Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329113 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5644f7de-07a8-4d2b-829b-d52210da47f5" containerName="extract" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329125 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5644f7de-07a8-4d2b-829b-d52210da47f5" containerName="extract" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329138 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54dd00c7-7600-40e0-9053-4f151bea65cd" containerName="util" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329144 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="54dd00c7-7600-40e0-9053-4f151bea65cd" containerName="util" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329150 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ddd4ed-1f36-4839-a9a2-edb28d549c98" containerName="pull" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329156 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ddd4ed-1f36-4839-a9a2-edb28d549c98" containerName="pull" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329171 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54dd00c7-7600-40e0-9053-4f151bea65cd" containerName="pull" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329177 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="54dd00c7-7600-40e0-9053-4f151bea65cd" containerName="pull" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329185 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54dd00c7-7600-40e0-9053-4f151bea65cd" containerName="extract" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329190 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="54dd00c7-7600-40e0-9053-4f151bea65cd" containerName="extract" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329196 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5644f7de-07a8-4d2b-829b-d52210da47f5" containerName="util" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329201 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5644f7de-07a8-4d2b-829b-d52210da47f5" containerName="util" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329207 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ddd4ed-1f36-4839-a9a2-edb28d549c98" containerName="extract" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329212 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ddd4ed-1f36-4839-a9a2-edb28d549c98" containerName="extract" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329218 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5644f7de-07a8-4d2b-829b-d52210da47f5" containerName="pull" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329223 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5644f7de-07a8-4d2b-829b-d52210da47f5" containerName="pull" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329230 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b9de7e-f20d-415d-861d-ff5aff65c788" containerName="util" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329235 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b9de7e-f20d-415d-861d-ff5aff65c788" containerName="util" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329241 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b9de7e-f20d-415d-861d-ff5aff65c788" containerName="extract" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329247 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b9de7e-f20d-415d-861d-ff5aff65c788" containerName="extract" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329253 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ddd4ed-1f36-4839-a9a2-edb28d549c98" containerName="util" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329259 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ddd4ed-1f36-4839-a9a2-edb28d549c98" containerName="util" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329265 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b9de7e-f20d-415d-861d-ff5aff65c788" containerName="pull" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329269 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b9de7e-f20d-415d-861d-ff5aff65c788" containerName="pull" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329318 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="54dd00c7-7600-40e0-9053-4f151bea65cd" containerName="extract" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329324 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5644f7de-07a8-4d2b-829b-d52210da47f5" containerName="extract" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329332 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89ddd4ed-1f36-4839-a9a2-edb28d549c98" containerName="extract" Jul 21 12:34:46.329385 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.329342 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b9de7e-f20d-415d-861d-ff5aff65c788" containerName="extract" Jul 21 12:34:46.333695 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.333676 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb" Jul 21 12:34:46.336784 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.336757 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 12:34:46.338333 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.338314 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d4b6n\"" Jul 21 12:34:46.338565 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.338542 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 12:34:46.356656 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.356624 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb"] Jul 21 12:34:46.411628 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.411590 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvgxb\" (UniqueName: \"kubernetes.io/projected/37d96f60-c449-434a-b0a3-f6525c812a23-kube-api-access-rvgxb\") pod \"kuadrant-operator-controller-manager-86878c769f-sgwcb\" (UID: \"37d96f60-c449-434a-b0a3-f6525c812a23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb" Jul 21 12:34:46.411829 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.411639 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37d96f60-c449-434a-b0a3-f6525c812a23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sgwcb\" (UID: \"37d96f60-c449-434a-b0a3-f6525c812a23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb" Jul 21 12:34:46.420216 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.420183 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb"] Jul 21 12:34:46.420523 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:34:46.420498 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-rvgxb], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb" podUID="37d96f60-c449-434a-b0a3-f6525c812a23" Jul 21 12:34:46.433725 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.433682 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb"] Jul 21 12:34:46.512699 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.512667 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvgxb\" (UniqueName: \"kubernetes.io/projected/37d96f60-c449-434a-b0a3-f6525c812a23-kube-api-access-rvgxb\") pod \"kuadrant-operator-controller-manager-86878c769f-sgwcb\" (UID: \"37d96f60-c449-434a-b0a3-f6525c812a23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb" Jul 21 12:34:46.512907 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.512715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37d96f60-c449-434a-b0a3-f6525c812a23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sgwcb\" (UID: \"37d96f60-c449-434a-b0a3-f6525c812a23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb" Jul 21 12:34:46.513101 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.513084 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37d96f60-c449-434a-b0a3-f6525c812a23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sgwcb\" (UID: \"37d96f60-c449-434a-b0a3-f6525c812a23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb" Jul 21 12:34:46.515418 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:34:46.515388 2574 projected.go:196] Error preparing data for projected volume kube-api-access-rvgxb for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb: failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-138-11.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-138-11.ec2.internal' and this object Jul 21 12:34:46.515539 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:34:46.515486 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37d96f60-c449-434a-b0a3-f6525c812a23-kube-api-access-rvgxb podName:37d96f60-c449-434a-b0a3-f6525c812a23 nodeName:}" failed. No retries permitted until 2026-07-21 12:34:47.01546274 +0000 UTC m=+433.665876095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rvgxb" (UniqueName: "kubernetes.io/projected/37d96f60-c449-434a-b0a3-f6525c812a23-kube-api-access-rvgxb") pod "kuadrant-operator-controller-manager-86878c769f-sgwcb" (UID: "37d96f60-c449-434a-b0a3-f6525c812a23") : failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-138-11.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-138-11.ec2.internal' and this object Jul 21 12:34:46.976832 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.976800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb" Jul 21 12:34:46.979816 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:34:46.979771 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-sgwcb\" is forbidden: User \"system:node:ip-10-0-138-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-11.ec2.internal' and this object" podUID="37d96f60-c449-434a-b0a3-f6525c812a23" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb" Jul 21 12:34:46.984119 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:46.984096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb" Jul 21 12:34:46.987610 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:34:46.987586 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-sgwcb\" is forbidden: User \"system:node:ip-10-0-138-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-11.ec2.internal' and this object" podUID="37d96f60-c449-434a-b0a3-f6525c812a23" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb" Jul 21 12:34:47.017152 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:47.017117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvgxb\" (UniqueName: \"kubernetes.io/projected/37d96f60-c449-434a-b0a3-f6525c812a23-kube-api-access-rvgxb\") pod \"kuadrant-operator-controller-manager-86878c769f-sgwcb\" (UID: \"37d96f60-c449-434a-b0a3-f6525c812a23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb" Jul 21 12:34:47.020047 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:34:47.020028 2574 projected.go:196] Error preparing data for projected volume kube-api-access-rvgxb for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb: failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-138-11.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-138-11.ec2.internal' and this object Jul 21 12:34:47.020131 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:34:47.020120 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37d96f60-c449-434a-b0a3-f6525c812a23-kube-api-access-rvgxb podName:37d96f60-c449-434a-b0a3-f6525c812a23 nodeName:}" failed. No retries permitted until 2026-07-21 12:34:48.02010311 +0000 UTC m=+434.670516448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rvgxb" (UniqueName: "kubernetes.io/projected/37d96f60-c449-434a-b0a3-f6525c812a23-kube-api-access-rvgxb") pod "kuadrant-operator-controller-manager-86878c769f-sgwcb" (UID: "37d96f60-c449-434a-b0a3-f6525c812a23") : failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-138-11.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-138-11.ec2.internal' and this object Jul 21 12:34:47.117786 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:47.117740 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37d96f60-c449-434a-b0a3-f6525c812a23-extensions-socket-volume\") pod \"37d96f60-c449-434a-b0a3-f6525c812a23\" (UID: \"37d96f60-c449-434a-b0a3-f6525c812a23\") " Jul 21 12:34:47.117989 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:47.117912 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvgxb\" (UniqueName: \"kubernetes.io/projected/37d96f60-c449-434a-b0a3-f6525c812a23-kube-api-access-rvgxb\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:34:47.118088 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:47.118063 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d96f60-c449-434a-b0a3-f6525c812a23-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "37d96f60-c449-434a-b0a3-f6525c812a23" (UID: "37d96f60-c449-434a-b0a3-f6525c812a23"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:34:47.218881 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:47.218821 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37d96f60-c449-434a-b0a3-f6525c812a23-extensions-socket-volume\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:34:47.925069 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:47.925034 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37d96f60-c449-434a-b0a3-f6525c812a23" path="/var/lib/kubelet/pods/37d96f60-c449-434a-b0a3-f6525c812a23/volumes" Jul 21 12:34:47.982065 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:34:47.982031 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb" Jul 21 12:34:47.986251 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:34:47.986216 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-sgwcb\" is forbidden: User \"system:node:ip-10-0-138-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-11.ec2.internal' and this object" podUID="37d96f60-c449-434a-b0a3-f6525c812a23" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sgwcb" Jul 21 12:35:31.338168 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.338130 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sjj7d"] Jul 21 12:35:31.341376 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.341358 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" Jul 21 12:35:31.344063 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.344041 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 12:35:31.344317 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.344298 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 12:35:31.345244 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.345224 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 12:35:31.345325 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.345306 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ckjqb\"" Jul 21 12:35:31.351983 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.351956 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sjj7d"] Jul 21 12:35:31.397258 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.397219 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f8d8a51-0664-44bd-ab96-0bb72202d81e-config-file\") pod \"limitador-limitador-64c94fd64f-sjj7d\" (UID: \"8f8d8a51-0664-44bd-ab96-0bb72202d81e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" Jul 21 12:35:31.397258 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.397262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb55m\" (UniqueName: \"kubernetes.io/projected/8f8d8a51-0664-44bd-ab96-0bb72202d81e-kube-api-access-sb55m\") pod \"limitador-limitador-64c94fd64f-sjj7d\" (UID: \"8f8d8a51-0664-44bd-ab96-0bb72202d81e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" Jul 21 12:35:31.438114 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.438081 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sjj7d"] Jul 21 12:35:31.498725 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.498689 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f8d8a51-0664-44bd-ab96-0bb72202d81e-config-file\") pod \"limitador-limitador-64c94fd64f-sjj7d\" (UID: \"8f8d8a51-0664-44bd-ab96-0bb72202d81e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" Jul 21 12:35:31.498725 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.498727 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb55m\" (UniqueName: \"kubernetes.io/projected/8f8d8a51-0664-44bd-ab96-0bb72202d81e-kube-api-access-sb55m\") pod \"limitador-limitador-64c94fd64f-sjj7d\" (UID: \"8f8d8a51-0664-44bd-ab96-0bb72202d81e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" Jul 21 12:35:31.499442 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.499416 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f8d8a51-0664-44bd-ab96-0bb72202d81e-config-file\") pod \"limitador-limitador-64c94fd64f-sjj7d\" (UID: \"8f8d8a51-0664-44bd-ab96-0bb72202d81e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" Jul 21 12:35:31.508088 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.508061 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb55m\" (UniqueName: \"kubernetes.io/projected/8f8d8a51-0664-44bd-ab96-0bb72202d81e-kube-api-access-sb55m\") pod \"limitador-limitador-64c94fd64f-sjj7d\" (UID: \"8f8d8a51-0664-44bd-ab96-0bb72202d81e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" Jul 21 12:35:31.652129 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.652034 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" Jul 21 12:35:31.796304 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:31.795969 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sjj7d"] Jul 21 12:35:32.200655 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:32.200618 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" event={"ID":"8f8d8a51-0664-44bd-ab96-0bb72202d81e","Type":"ContainerStarted","Data":"23470c9b8706274872a0b968befc7bacc566e292c67e100ba6351d114a007076"} Jul 21 12:35:35.219424 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:35.219328 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" event={"ID":"8f8d8a51-0664-44bd-ab96-0bb72202d81e","Type":"ContainerStarted","Data":"b70bfd180fad89200b5f054072f5c3ad9ead7a5ebdcff2c3234d37a50fe326b7"} Jul 21 12:35:35.219808 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:35.219467 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" Jul 21 12:35:35.237978 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:35.237904 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" podStartSLOduration=1.251509553 podStartE2EDuration="4.237887383s" totalImagesPullingTime="2.98637783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:35:35.237272492 +0000 UTC m=+481.887685865" watchObservedRunningTime="2026-07-21 12:35:35.237887383 +0000 UTC m=+481.888300745" Jul 21 12:35:46.231854 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:46.231819 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" Jul 21 12:35:48.393595 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:48.393558 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sjj7d"] Jul 21 12:35:48.394010 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:48.393778 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" podUID="8f8d8a51-0664-44bd-ab96-0bb72202d81e" containerName="limitador" containerID="cri-o://b70bfd180fad89200b5f054072f5c3ad9ead7a5ebdcff2c3234d37a50fe326b7" gracePeriod=30 Jul 21 12:35:48.941691 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:48.941663 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" Jul 21 12:35:49.057050 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.057015 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb55m\" (UniqueName: \"kubernetes.io/projected/8f8d8a51-0664-44bd-ab96-0bb72202d81e-kube-api-access-sb55m\") pod \"8f8d8a51-0664-44bd-ab96-0bb72202d81e\" (UID: \"8f8d8a51-0664-44bd-ab96-0bb72202d81e\") " Jul 21 12:35:49.057257 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.057095 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f8d8a51-0664-44bd-ab96-0bb72202d81e-config-file\") pod \"8f8d8a51-0664-44bd-ab96-0bb72202d81e\" (UID: \"8f8d8a51-0664-44bd-ab96-0bb72202d81e\") " Jul 21 12:35:49.057461 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.057432 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8d8a51-0664-44bd-ab96-0bb72202d81e-config-file" (OuterVolumeSpecName: "config-file") pod "8f8d8a51-0664-44bd-ab96-0bb72202d81e" (UID: "8f8d8a51-0664-44bd-ab96-0bb72202d81e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:35:49.059246 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.059212 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8d8a51-0664-44bd-ab96-0bb72202d81e-kube-api-access-sb55m" (OuterVolumeSpecName: "kube-api-access-sb55m") pod "8f8d8a51-0664-44bd-ab96-0bb72202d81e" (UID: "8f8d8a51-0664-44bd-ab96-0bb72202d81e"). InnerVolumeSpecName "kube-api-access-sb55m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:35:49.157645 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.157609 2574 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f8d8a51-0664-44bd-ab96-0bb72202d81e-config-file\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:35:49.157645 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.157639 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb55m\" (UniqueName: \"kubernetes.io/projected/8f8d8a51-0664-44bd-ab96-0bb72202d81e-kube-api-access-sb55m\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:35:49.297232 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.297198 2574 generic.go:356] "Generic (PLEG): container finished" podID="8f8d8a51-0664-44bd-ab96-0bb72202d81e" containerID="b70bfd180fad89200b5f054072f5c3ad9ead7a5ebdcff2c3234d37a50fe326b7" exitCode=0 Jul 21 12:35:49.297415 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.297245 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" Jul 21 12:35:49.297415 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.297289 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" event={"ID":"8f8d8a51-0664-44bd-ab96-0bb72202d81e","Type":"ContainerDied","Data":"b70bfd180fad89200b5f054072f5c3ad9ead7a5ebdcff2c3234d37a50fe326b7"} Jul 21 12:35:49.297415 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.297344 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-sjj7d" event={"ID":"8f8d8a51-0664-44bd-ab96-0bb72202d81e","Type":"ContainerDied","Data":"23470c9b8706274872a0b968befc7bacc566e292c67e100ba6351d114a007076"} Jul 21 12:35:49.297415 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.297368 2574 scope.go:117] "RemoveContainer" containerID="b70bfd180fad89200b5f054072f5c3ad9ead7a5ebdcff2c3234d37a50fe326b7" Jul 21 12:35:49.307002 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.306980 2574 scope.go:117] "RemoveContainer" containerID="b70bfd180fad89200b5f054072f5c3ad9ead7a5ebdcff2c3234d37a50fe326b7" Jul 21 12:35:49.307314 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:35:49.307267 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70bfd180fad89200b5f054072f5c3ad9ead7a5ebdcff2c3234d37a50fe326b7\": container with ID starting with b70bfd180fad89200b5f054072f5c3ad9ead7a5ebdcff2c3234d37a50fe326b7 not found: ID does not exist" containerID="b70bfd180fad89200b5f054072f5c3ad9ead7a5ebdcff2c3234d37a50fe326b7" Jul 21 12:35:49.307314 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.307297 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70bfd180fad89200b5f054072f5c3ad9ead7a5ebdcff2c3234d37a50fe326b7"} err="failed to get container status \"b70bfd180fad89200b5f054072f5c3ad9ead7a5ebdcff2c3234d37a50fe326b7\": rpc error: code = NotFound desc = could not find container \"b70bfd180fad89200b5f054072f5c3ad9ead7a5ebdcff2c3234d37a50fe326b7\": container with ID starting with b70bfd180fad89200b5f054072f5c3ad9ead7a5ebdcff2c3234d37a50fe326b7 not found: ID does not exist" Jul 21 12:35:49.326558 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.326523 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sjj7d"] Jul 21 12:35:49.331828 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.331798 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sjj7d"] Jul 21 12:35:49.924853 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:49.924803 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f8d8a51-0664-44bd-ab96-0bb72202d81e" path="/var/lib/kubelet/pods/8f8d8a51-0664-44bd-ab96-0bb72202d81e/volumes" Jul 21 12:35:53.122418 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.122382 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk"] Jul 21 12:35:53.122916 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.122856 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8d8a51-0664-44bd-ab96-0bb72202d81e" containerName="limitador" Jul 21 12:35:53.122916 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.122873 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8d8a51-0664-44bd-ab96-0bb72202d81e" containerName="limitador" Jul 21 12:35:53.123070 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.122997 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8d8a51-0664-44bd-ab96-0bb72202d81e" containerName="limitador" Jul 21 12:35:53.129192 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.129166 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk" Jul 21 12:35:53.136138 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.136117 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 21 12:35:53.136271 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.136251 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 12:35:53.137230 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.137215 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 12:35:53.137315 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.137217 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-t7vc6\"" Jul 21 12:35:53.142309 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.142280 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk"] Jul 21 12:35:53.195925 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.195885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6ff522d-fdc1-4d74-9ec0-e06665f17706-data\") pod \"postgres-69fb4594cf-rbzxk\" (UID: \"d6ff522d-fdc1-4d74-9ec0-e06665f17706\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk" Jul 21 12:35:53.196110 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.196065 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njx8\" (UniqueName: \"kubernetes.io/projected/d6ff522d-fdc1-4d74-9ec0-e06665f17706-kube-api-access-9njx8\") pod \"postgres-69fb4594cf-rbzxk\" (UID: \"d6ff522d-fdc1-4d74-9ec0-e06665f17706\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk" Jul 21 12:35:53.296622 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.296585 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njx8\" (UniqueName: \"kubernetes.io/projected/d6ff522d-fdc1-4d74-9ec0-e06665f17706-kube-api-access-9njx8\") pod \"postgres-69fb4594cf-rbzxk\" (UID: \"d6ff522d-fdc1-4d74-9ec0-e06665f17706\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk" Jul 21 12:35:53.296818 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.296711 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6ff522d-fdc1-4d74-9ec0-e06665f17706-data\") pod \"postgres-69fb4594cf-rbzxk\" (UID: \"d6ff522d-fdc1-4d74-9ec0-e06665f17706\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk" Jul 21 12:35:53.297112 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.297091 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6ff522d-fdc1-4d74-9ec0-e06665f17706-data\") pod \"postgres-69fb4594cf-rbzxk\" (UID: \"d6ff522d-fdc1-4d74-9ec0-e06665f17706\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk" Jul 21 12:35:53.305052 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.305023 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njx8\" (UniqueName: \"kubernetes.io/projected/d6ff522d-fdc1-4d74-9ec0-e06665f17706-kube-api-access-9njx8\") pod \"postgres-69fb4594cf-rbzxk\" (UID: \"d6ff522d-fdc1-4d74-9ec0-e06665f17706\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk" Jul 21 12:35:53.439301 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.439201 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk" Jul 21 12:35:53.585407 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:53.585379 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk"] Jul 21 12:35:53.587582 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:35:53.587554 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ff522d_fdc1_4d74_9ec0_e06665f17706.slice/crio-17100c025c2f3b6459442cbd5d1a4a2b2630c5bb5f80786adfab424f13254ed2 WatchSource:0}: Error finding container 17100c025c2f3b6459442cbd5d1a4a2b2630c5bb5f80786adfab424f13254ed2: Status 404 returned error can't find the container with id 17100c025c2f3b6459442cbd5d1a4a2b2630c5bb5f80786adfab424f13254ed2 Jul 21 12:35:54.328580 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:54.328534 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk" event={"ID":"d6ff522d-fdc1-4d74-9ec0-e06665f17706","Type":"ContainerStarted","Data":"17100c025c2f3b6459442cbd5d1a4a2b2630c5bb5f80786adfab424f13254ed2"} Jul 21 12:35:58.353438 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:58.353402 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk" event={"ID":"d6ff522d-fdc1-4d74-9ec0-e06665f17706","Type":"ContainerStarted","Data":"0b5206956e652b3cd33090c21af88cce98dff7de882fb4087efe2fa7c833931b"} Jul 21 12:35:58.353807 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:58.353498 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk" Jul 21 12:35:58.387305 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:35:58.387254 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk" podStartSLOduration=0.711325979 podStartE2EDuration="5.387237189s" totalImagesPullingTime="4.67591121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:35:58.38616747 +0000 UTC m=+505.036580833" watchObservedRunningTime="2026-07-21 12:35:58.387237189 +0000 UTC m=+505.037650550" Jul 21 12:36:04.388758 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:04.388729 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rbzxk" Jul 21 12:36:11.869284 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:11.869241 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f6fb5dfcb-k2rh6"] Jul 21 12:36:11.872842 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:11.872824 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" Jul 21 12:36:11.876066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:11.876043 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 12:36:11.876066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:11.876062 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 12:36:11.876262 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:11.876161 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6jfm5\"" Jul 21 12:36:11.883049 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:11.883021 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f6fb5dfcb-k2rh6"] Jul 21 12:36:11.972362 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:11.972325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s22fb\" (UniqueName: \"kubernetes.io/projected/ce25736d-1ec8-4d63-a785-fed28169ce04-kube-api-access-s22fb\") pod \"maas-controller-6f6fb5dfcb-k2rh6\" (UID: \"ce25736d-1ec8-4d63-a785-fed28169ce04\") " pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" Jul 21 12:36:11.972566 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:11.972385 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce25736d-1ec8-4d63-a785-fed28169ce04-webhook-cert\") pod \"maas-controller-6f6fb5dfcb-k2rh6\" (UID: \"ce25736d-1ec8-4d63-a785-fed28169ce04\") " pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" Jul 21 12:36:12.041527 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.041495 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-jkqjx"] Jul 21 12:36:12.044838 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.044816 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" Jul 21 12:36:12.056083 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.056055 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-jkqjx"] Jul 21 12:36:12.073187 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.073145 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce25736d-1ec8-4d63-a785-fed28169ce04-webhook-cert\") pod \"maas-controller-6f6fb5dfcb-k2rh6\" (UID: \"ce25736d-1ec8-4d63-a785-fed28169ce04\") " pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" Jul 21 12:36:12.073379 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.073265 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s22fb\" (UniqueName: \"kubernetes.io/projected/ce25736d-1ec8-4d63-a785-fed28169ce04-kube-api-access-s22fb\") pod \"maas-controller-6f6fb5dfcb-k2rh6\" (UID: \"ce25736d-1ec8-4d63-a785-fed28169ce04\") " pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" Jul 21 12:36:12.073379 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:36:12.073296 2574 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 21 12:36:12.073379 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:36:12.073377 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce25736d-1ec8-4d63-a785-fed28169ce04-webhook-cert podName:ce25736d-1ec8-4d63-a785-fed28169ce04 nodeName:}" failed. No retries permitted until 2026-07-21 12:36:12.573356743 +0000 UTC m=+519.223770095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/ce25736d-1ec8-4d63-a785-fed28169ce04-webhook-cert") pod "maas-controller-6f6fb5dfcb-k2rh6" (UID: "ce25736d-1ec8-4d63-a785-fed28169ce04") : secret "maas-controller-webhook-cert" not found Jul 21 12:36:12.082206 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.082163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s22fb\" (UniqueName: \"kubernetes.io/projected/ce25736d-1ec8-4d63-a785-fed28169ce04-kube-api-access-s22fb\") pod \"maas-controller-6f6fb5dfcb-k2rh6\" (UID: \"ce25736d-1ec8-4d63-a785-fed28169ce04\") " pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" Jul 21 12:36:12.173843 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.173738 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54d35ea9-b8ea-49bc-b497-8eaec0d39a0a-webhook-cert\") pod \"maas-controller-68fb8f6cb6-jkqjx\" (UID: \"54d35ea9-b8ea-49bc-b497-8eaec0d39a0a\") " pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" Jul 21 12:36:12.173843 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.173789 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg4mh\" (UniqueName: \"kubernetes.io/projected/54d35ea9-b8ea-49bc-b497-8eaec0d39a0a-kube-api-access-lg4mh\") pod \"maas-controller-68fb8f6cb6-jkqjx\" (UID: \"54d35ea9-b8ea-49bc-b497-8eaec0d39a0a\") " pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" Jul 21 12:36:12.274582 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.274541 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54d35ea9-b8ea-49bc-b497-8eaec0d39a0a-webhook-cert\") pod \"maas-controller-68fb8f6cb6-jkqjx\" (UID: \"54d35ea9-b8ea-49bc-b497-8eaec0d39a0a\") " pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" Jul 21 12:36:12.274792 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.274593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg4mh\" (UniqueName: \"kubernetes.io/projected/54d35ea9-b8ea-49bc-b497-8eaec0d39a0a-kube-api-access-lg4mh\") pod \"maas-controller-68fb8f6cb6-jkqjx\" (UID: \"54d35ea9-b8ea-49bc-b497-8eaec0d39a0a\") " pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" Jul 21 12:36:12.277045 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.277021 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54d35ea9-b8ea-49bc-b497-8eaec0d39a0a-webhook-cert\") pod \"maas-controller-68fb8f6cb6-jkqjx\" (UID: \"54d35ea9-b8ea-49bc-b497-8eaec0d39a0a\") " pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" Jul 21 12:36:12.282763 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.282741 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg4mh\" (UniqueName: \"kubernetes.io/projected/54d35ea9-b8ea-49bc-b497-8eaec0d39a0a-kube-api-access-lg4mh\") pod \"maas-controller-68fb8f6cb6-jkqjx\" (UID: \"54d35ea9-b8ea-49bc-b497-8eaec0d39a0a\") " pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" Jul 21 12:36:12.355870 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.355830 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" Jul 21 12:36:12.496300 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.496274 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-jkqjx"] Jul 21 12:36:12.498568 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:36:12.498542 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d35ea9_b8ea_49bc_b497_8eaec0d39a0a.slice/crio-72546cf206cbf80b08016faf5e162ec932f11ca3151003b974e563c0bd270703 WatchSource:0}: Error finding container 72546cf206cbf80b08016faf5e162ec932f11ca3151003b974e563c0bd270703: Status 404 returned error can't find the container with id 72546cf206cbf80b08016faf5e162ec932f11ca3151003b974e563c0bd270703 Jul 21 12:36:12.577090 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.577047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce25736d-1ec8-4d63-a785-fed28169ce04-webhook-cert\") pod \"maas-controller-6f6fb5dfcb-k2rh6\" (UID: \"ce25736d-1ec8-4d63-a785-fed28169ce04\") " pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" Jul 21 12:36:12.579493 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.579472 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce25736d-1ec8-4d63-a785-fed28169ce04-webhook-cert\") pod \"maas-controller-6f6fb5dfcb-k2rh6\" (UID: \"ce25736d-1ec8-4d63-a785-fed28169ce04\") " pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" Jul 21 12:36:12.784246 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.784200 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" Jul 21 12:36:12.921254 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:12.921229 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f6fb5dfcb-k2rh6"] Jul 21 12:36:12.923094 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:36:12.923065 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce25736d_1ec8_4d63_a785_fed28169ce04.slice/crio-f2c07c02e60b2c7b1ba1f3b5647cb7184a3565b87f4f1f746237fb32f22a0485 WatchSource:0}: Error finding container f2c07c02e60b2c7b1ba1f3b5647cb7184a3565b87f4f1f746237fb32f22a0485: Status 404 returned error can't find the container with id f2c07c02e60b2c7b1ba1f3b5647cb7184a3565b87f4f1f746237fb32f22a0485 Jul 21 12:36:13.434357 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:13.434273 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" event={"ID":"54d35ea9-b8ea-49bc-b497-8eaec0d39a0a","Type":"ContainerStarted","Data":"72546cf206cbf80b08016faf5e162ec932f11ca3151003b974e563c0bd270703"} Jul 21 12:36:13.436529 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:13.436452 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" event={"ID":"ce25736d-1ec8-4d63-a785-fed28169ce04","Type":"ContainerStarted","Data":"f2c07c02e60b2c7b1ba1f3b5647cb7184a3565b87f4f1f746237fb32f22a0485"} Jul 21 12:36:15.450281 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:15.450241 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" event={"ID":"54d35ea9-b8ea-49bc-b497-8eaec0d39a0a","Type":"ContainerStarted","Data":"40f56b106a3ae8fd798498edb5aac8cea9e3490ca4f33646996134ea4427c5fe"} Jul 21 12:36:15.450728 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:15.450465 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" Jul 21 12:36:15.451842 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:15.451815 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" event={"ID":"ce25736d-1ec8-4d63-a785-fed28169ce04","Type":"ContainerStarted","Data":"e7cc6831d0aa8465047b96eaaf28dc48037e9463f99d137a540c0c52db8ff07d"} Jul 21 12:36:15.451968 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:15.451935 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" Jul 21 12:36:15.470515 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:15.470460 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" podStartSLOduration=0.980498479 podStartE2EDuration="3.470438797s" totalImagesPullingTime="2.489940318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:36:15.46844291 +0000 UTC m=+522.118856295" watchObservedRunningTime="2026-07-21 12:36:15.470438797 +0000 UTC m=+522.120852158" Jul 21 12:36:15.489591 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:15.489537 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" podStartSLOduration=2.417581433 podStartE2EDuration="4.489520951s" totalImagesPullingTime="2.071939518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:36:15.487246673 +0000 UTC m=+522.137660033" watchObservedRunningTime="2026-07-21 12:36:15.489520951 +0000 UTC m=+522.139934312" Jul 21 12:36:26.458886 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:26.458807 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" Jul 21 12:36:26.458886 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:26.458879 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" Jul 21 12:36:26.523654 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:26.523620 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f6fb5dfcb-k2rh6"] Jul 21 12:36:26.524024 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:26.523993 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" podUID="ce25736d-1ec8-4d63-a785-fed28169ce04" containerName="manager" containerID="cri-o://e7cc6831d0aa8465047b96eaaf28dc48037e9463f99d137a540c0c52db8ff07d" gracePeriod=30 Jul 21 12:36:26.776897 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:26.776871 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" Jul 21 12:36:26.782924 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:26.782904 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce25736d-1ec8-4d63-a785-fed28169ce04-webhook-cert\") pod \"ce25736d-1ec8-4d63-a785-fed28169ce04\" (UID: \"ce25736d-1ec8-4d63-a785-fed28169ce04\") " Jul 21 12:36:26.783008 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:26.782996 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s22fb\" (UniqueName: \"kubernetes.io/projected/ce25736d-1ec8-4d63-a785-fed28169ce04-kube-api-access-s22fb\") pod \"ce25736d-1ec8-4d63-a785-fed28169ce04\" (UID: \"ce25736d-1ec8-4d63-a785-fed28169ce04\") " Jul 21 12:36:26.785098 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:26.785062 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce25736d-1ec8-4d63-a785-fed28169ce04-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ce25736d-1ec8-4d63-a785-fed28169ce04" (UID: "ce25736d-1ec8-4d63-a785-fed28169ce04"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:36:26.785098 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:26.785077 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce25736d-1ec8-4d63-a785-fed28169ce04-kube-api-access-s22fb" (OuterVolumeSpecName: "kube-api-access-s22fb") pod "ce25736d-1ec8-4d63-a785-fed28169ce04" (UID: "ce25736d-1ec8-4d63-a785-fed28169ce04"). InnerVolumeSpecName "kube-api-access-s22fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:36:26.884475 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:26.884433 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s22fb\" (UniqueName: \"kubernetes.io/projected/ce25736d-1ec8-4d63-a785-fed28169ce04-kube-api-access-s22fb\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:36:26.884475 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:26.884467 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce25736d-1ec8-4d63-a785-fed28169ce04-webhook-cert\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:36:27.519704 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:27.519667 2574 generic.go:356] "Generic (PLEG): container finished" podID="ce25736d-1ec8-4d63-a785-fed28169ce04" containerID="e7cc6831d0aa8465047b96eaaf28dc48037e9463f99d137a540c0c52db8ff07d" exitCode=0 Jul 21 12:36:27.520121 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:27.519713 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" Jul 21 12:36:27.520121 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:27.519749 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" event={"ID":"ce25736d-1ec8-4d63-a785-fed28169ce04","Type":"ContainerDied","Data":"e7cc6831d0aa8465047b96eaaf28dc48037e9463f99d137a540c0c52db8ff07d"} Jul 21 12:36:27.520121 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:27.519798 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f6fb5dfcb-k2rh6" event={"ID":"ce25736d-1ec8-4d63-a785-fed28169ce04","Type":"ContainerDied","Data":"f2c07c02e60b2c7b1ba1f3b5647cb7184a3565b87f4f1f746237fb32f22a0485"} Jul 21 12:36:27.520121 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:27.519821 2574 scope.go:117] "RemoveContainer" containerID="e7cc6831d0aa8465047b96eaaf28dc48037e9463f99d137a540c0c52db8ff07d" Jul 21 12:36:27.529403 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:27.529291 2574 scope.go:117] "RemoveContainer" containerID="e7cc6831d0aa8465047b96eaaf28dc48037e9463f99d137a540c0c52db8ff07d" Jul 21 12:36:27.529626 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:36:27.529604 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7cc6831d0aa8465047b96eaaf28dc48037e9463f99d137a540c0c52db8ff07d\": container with ID starting with e7cc6831d0aa8465047b96eaaf28dc48037e9463f99d137a540c0c52db8ff07d not found: ID does not exist" containerID="e7cc6831d0aa8465047b96eaaf28dc48037e9463f99d137a540c0c52db8ff07d" Jul 21 12:36:27.529664 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:27.529634 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7cc6831d0aa8465047b96eaaf28dc48037e9463f99d137a540c0c52db8ff07d"} err="failed to get container status \"e7cc6831d0aa8465047b96eaaf28dc48037e9463f99d137a540c0c52db8ff07d\": rpc error: code = NotFound desc = could not find container \"e7cc6831d0aa8465047b96eaaf28dc48037e9463f99d137a540c0c52db8ff07d\": container with ID starting with e7cc6831d0aa8465047b96eaaf28dc48037e9463f99d137a540c0c52db8ff07d not found: ID does not exist" Jul 21 12:36:27.541560 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:27.541528 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f6fb5dfcb-k2rh6"] Jul 21 12:36:27.545336 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:27.545307 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f6fb5dfcb-k2rh6"] Jul 21 12:36:27.924465 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:27.924384 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce25736d-1ec8-4d63-a785-fed28169ce04" path="/var/lib/kubelet/pods/ce25736d-1ec8-4d63-a785-fed28169ce04/volumes" Jul 21 12:36:46.920000 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:46.919964 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj"] Jul 21 12:36:46.920472 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:46.920310 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce25736d-1ec8-4d63-a785-fed28169ce04" containerName="manager" Jul 21 12:36:46.920472 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:46.920321 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce25736d-1ec8-4d63-a785-fed28169ce04" containerName="manager" Jul 21 12:36:46.920472 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:46.920383 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce25736d-1ec8-4d63-a785-fed28169ce04" containerName="manager" Jul 21 12:36:46.950600 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:46.950554 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj"] Jul 21 12:36:46.950761 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:46.950586 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" Jul 21 12:36:46.953738 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:46.953713 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 21 12:36:46.953899 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:46.953760 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zznvl\"" Jul 21 12:36:47.062713 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.062674 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6nhf\" (UniqueName: \"kubernetes.io/projected/cb66477c-21d2-47c5-a8ab-86a4a0a3bb35-kube-api-access-q6nhf\") pod \"maas-api-5cf648dd67-hqdgj\" (UID: \"cb66477c-21d2-47c5-a8ab-86a4a0a3bb35\") " pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" Jul 21 12:36:47.062902 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.062768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb66477c-21d2-47c5-a8ab-86a4a0a3bb35-maas-api-tls\") pod \"maas-api-5cf648dd67-hqdgj\" (UID: \"cb66477c-21d2-47c5-a8ab-86a4a0a3bb35\") " pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" Jul 21 12:36:47.062902 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.062785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb66477c-21d2-47c5-a8ab-86a4a0a3bb35-openshift-service-ca\") pod \"maas-api-5cf648dd67-hqdgj\" (UID: \"cb66477c-21d2-47c5-a8ab-86a4a0a3bb35\") " pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" Jul 21 12:36:47.164084 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.164048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nhf\" (UniqueName: \"kubernetes.io/projected/cb66477c-21d2-47c5-a8ab-86a4a0a3bb35-kube-api-access-q6nhf\") pod \"maas-api-5cf648dd67-hqdgj\" (UID: \"cb66477c-21d2-47c5-a8ab-86a4a0a3bb35\") " pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" Jul 21 12:36:47.164281 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.164133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb66477c-21d2-47c5-a8ab-86a4a0a3bb35-maas-api-tls\") pod \"maas-api-5cf648dd67-hqdgj\" (UID: \"cb66477c-21d2-47c5-a8ab-86a4a0a3bb35\") " pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" Jul 21 12:36:47.164281 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.164151 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb66477c-21d2-47c5-a8ab-86a4a0a3bb35-openshift-service-ca\") pod \"maas-api-5cf648dd67-hqdgj\" (UID: \"cb66477c-21d2-47c5-a8ab-86a4a0a3bb35\") " pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" Jul 21 12:36:47.164779 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.164757 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb66477c-21d2-47c5-a8ab-86a4a0a3bb35-openshift-service-ca\") pod \"maas-api-5cf648dd67-hqdgj\" (UID: \"cb66477c-21d2-47c5-a8ab-86a4a0a3bb35\") " pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" Jul 21 12:36:47.166630 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.166605 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb66477c-21d2-47c5-a8ab-86a4a0a3bb35-maas-api-tls\") pod \"maas-api-5cf648dd67-hqdgj\" (UID: \"cb66477c-21d2-47c5-a8ab-86a4a0a3bb35\") " pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" Jul 21 12:36:47.178451 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.178386 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6nhf\" (UniqueName: \"kubernetes.io/projected/cb66477c-21d2-47c5-a8ab-86a4a0a3bb35-kube-api-access-q6nhf\") pod \"maas-api-5cf648dd67-hqdgj\" (UID: \"cb66477c-21d2-47c5-a8ab-86a4a0a3bb35\") " pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" Jul 21 12:36:47.261726 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.261679 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" Jul 21 12:36:47.412507 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.412486 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj"] Jul 21 12:36:47.414872 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:36:47.414845 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb66477c_21d2_47c5_a8ab_86a4a0a3bb35.slice/crio-4b8b524cf7ecd4a925ed3c8ad41981215db2f03ae699a5cdcf1cafaea627251f WatchSource:0}: Error finding container 4b8b524cf7ecd4a925ed3c8ad41981215db2f03ae699a5cdcf1cafaea627251f: Status 404 returned error can't find the container with id 4b8b524cf7ecd4a925ed3c8ad41981215db2f03ae699a5cdcf1cafaea627251f Jul 21 12:36:47.550383 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.550349 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-d82c7"] Jul 21 12:36:47.554927 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.554902 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-d82c7" Jul 21 12:36:47.557576 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.557556 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 21 12:36:47.557636 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.557606 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-n7k64\"" Jul 21 12:36:47.563757 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.563732 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-d82c7"] Jul 21 12:36:47.567830 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.567799 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tczgm"] Jul 21 12:36:47.571226 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.571203 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-tczgm" Jul 21 12:36:47.586809 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.586771 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tczgm"] Jul 21 12:36:47.623792 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.623755 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" event={"ID":"cb66477c-21d2-47c5-a8ab-86a4a0a3bb35","Type":"ContainerStarted","Data":"4b8b524cf7ecd4a925ed3c8ad41981215db2f03ae699a5cdcf1cafaea627251f"} Jul 21 12:36:47.669374 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.669338 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb5c4cd-d237-4a89-b5a0-ee640f3afa27-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-d82c7\" (UID: \"8eb5c4cd-d237-4a89-b5a0-ee640f3afa27\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-d82c7" Jul 21 12:36:47.669555 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.669417 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612b3adb-d198-4182-a98e-78261ab1fa85-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tczgm\" (UID: \"612b3adb-d198-4182-a98e-78261ab1fa85\") " pod="openshift-ingress/payload-processing-5bf47cd988-tczgm" Jul 21 12:36:47.669555 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.669448 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6vdl\" (UniqueName: \"kubernetes.io/projected/612b3adb-d198-4182-a98e-78261ab1fa85-kube-api-access-d6vdl\") pod \"payload-processing-5bf47cd988-tczgm\" (UID: \"612b3adb-d198-4182-a98e-78261ab1fa85\") " pod="openshift-ingress/payload-processing-5bf47cd988-tczgm" Jul 21 12:36:47.669555 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.669473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr7dk\" (UniqueName: \"kubernetes.io/projected/8eb5c4cd-d237-4a89-b5a0-ee640f3afa27-kube-api-access-jr7dk\") pod \"payload-pre-processing-7677b774bb-d82c7\" (UID: \"8eb5c4cd-d237-4a89-b5a0-ee640f3afa27\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-d82c7" Jul 21 12:36:47.770010 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.769972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6vdl\" (UniqueName: \"kubernetes.io/projected/612b3adb-d198-4182-a98e-78261ab1fa85-kube-api-access-d6vdl\") pod \"payload-processing-5bf47cd988-tczgm\" (UID: \"612b3adb-d198-4182-a98e-78261ab1fa85\") " pod="openshift-ingress/payload-processing-5bf47cd988-tczgm" Jul 21 12:36:47.770199 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.770022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7dk\" (UniqueName: \"kubernetes.io/projected/8eb5c4cd-d237-4a89-b5a0-ee640f3afa27-kube-api-access-jr7dk\") pod \"payload-pre-processing-7677b774bb-d82c7\" (UID: \"8eb5c4cd-d237-4a89-b5a0-ee640f3afa27\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-d82c7" Jul 21 12:36:47.770199 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.770091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb5c4cd-d237-4a89-b5a0-ee640f3afa27-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-d82c7\" (UID: \"8eb5c4cd-d237-4a89-b5a0-ee640f3afa27\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-d82c7" Jul 21 12:36:47.770199 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.770148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612b3adb-d198-4182-a98e-78261ab1fa85-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tczgm\" (UID: \"612b3adb-d198-4182-a98e-78261ab1fa85\") " pod="openshift-ingress/payload-processing-5bf47cd988-tczgm" Jul 21 12:36:47.770994 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.770973 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612b3adb-d198-4182-a98e-78261ab1fa85-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tczgm\" (UID: \"612b3adb-d198-4182-a98e-78261ab1fa85\") " pod="openshift-ingress/payload-processing-5bf47cd988-tczgm" Jul 21 12:36:47.771075 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.770988 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb5c4cd-d237-4a89-b5a0-ee640f3afa27-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-d82c7\" (UID: \"8eb5c4cd-d237-4a89-b5a0-ee640f3afa27\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-d82c7" Jul 21 12:36:47.779418 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.779391 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7dk\" (UniqueName: \"kubernetes.io/projected/8eb5c4cd-d237-4a89-b5a0-ee640f3afa27-kube-api-access-jr7dk\") pod \"payload-pre-processing-7677b774bb-d82c7\" (UID: \"8eb5c4cd-d237-4a89-b5a0-ee640f3afa27\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-d82c7" Jul 21 12:36:47.779531 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.779423 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6vdl\" (UniqueName: \"kubernetes.io/projected/612b3adb-d198-4182-a98e-78261ab1fa85-kube-api-access-d6vdl\") pod \"payload-processing-5bf47cd988-tczgm\" (UID: \"612b3adb-d198-4182-a98e-78261ab1fa85\") " pod="openshift-ingress/payload-processing-5bf47cd988-tczgm" Jul 21 12:36:47.866635 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.866522 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-d82c7" Jul 21 12:36:47.881923 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:47.881892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-tczgm" Jul 21 12:36:48.040126 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:48.040100 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-d82c7"] Jul 21 12:36:48.040814 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:36:48.040788 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb5c4cd_d237_4a89_b5a0_ee640f3afa27.slice/crio-5c90225f92c83457d5fd67b80f44be5e4eaf84677dc590a9ab912f282d0b4523 WatchSource:0}: Error finding container 5c90225f92c83457d5fd67b80f44be5e4eaf84677dc590a9ab912f282d0b4523: Status 404 returned error can't find the container with id 5c90225f92c83457d5fd67b80f44be5e4eaf84677dc590a9ab912f282d0b4523 Jul 21 12:36:48.085416 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:48.085389 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tczgm"] Jul 21 12:36:48.086788 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:36:48.086759 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612b3adb_d198_4182_a98e_78261ab1fa85.slice/crio-399a34e090fbe282782fc4ceb688636a8b0f9ff108cdc59758b9dc5311d76750 WatchSource:0}: Error finding container 399a34e090fbe282782fc4ceb688636a8b0f9ff108cdc59758b9dc5311d76750: Status 404 returned error can't find the container with id 399a34e090fbe282782fc4ceb688636a8b0f9ff108cdc59758b9dc5311d76750 Jul 21 12:36:48.631260 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:48.631213 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-d82c7" event={"ID":"8eb5c4cd-d237-4a89-b5a0-ee640f3afa27","Type":"ContainerStarted","Data":"5c90225f92c83457d5fd67b80f44be5e4eaf84677dc590a9ab912f282d0b4523"} Jul 21 12:36:48.632584 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:48.632549 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-tczgm" event={"ID":"612b3adb-d198-4182-a98e-78261ab1fa85","Type":"ContainerStarted","Data":"399a34e090fbe282782fc4ceb688636a8b0f9ff108cdc59758b9dc5311d76750"} Jul 21 12:36:49.642654 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:49.642615 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" event={"ID":"cb66477c-21d2-47c5-a8ab-86a4a0a3bb35","Type":"ContainerStarted","Data":"ef482c6c448b6ea2cab1a50016ea6e6cbc9151efbcd90a91eaa2bd56f088ed53"} Jul 21 12:36:49.643122 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:49.642913 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" Jul 21 12:36:49.666065 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:49.665215 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" podStartSLOduration=1.969654154 podStartE2EDuration="3.665198963s" totalImagesPullingTime="1.695544809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:36:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:36:49.663159245 +0000 UTC m=+556.313572606" watchObservedRunningTime="2026-07-21 12:36:49.665198963 +0000 UTC m=+556.315612324" Jul 21 12:36:50.653549 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:50.653509 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-d82c7" event={"ID":"8eb5c4cd-d237-4a89-b5a0-ee640f3afa27","Type":"ContainerStarted","Data":"0faaca4f7a82a6d011ea78b29f38d864990c4bcb874f879b0e2793f15305a2d5"} Jul 21 12:36:50.654046 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:50.653621 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-d82c7" Jul 21 12:36:50.655386 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:50.655348 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-tczgm" event={"ID":"612b3adb-d198-4182-a98e-78261ab1fa85","Type":"ContainerStarted","Data":"00f2969a9a71c99e6be08cb578fb2a22342baf3afa0a315f311696b85434fd56"} Jul 21 12:36:50.655584 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:50.655567 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-tczgm" Jul 21 12:36:50.676395 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:50.676338 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-d82c7" podStartSLOduration=1.688876305 podStartE2EDuration="3.676321704s" totalImagesPullingTime="1.987445399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:36:50.673786424 +0000 UTC m=+557.324199786" watchObservedRunningTime="2026-07-21 12:36:50.676321704 +0000 UTC m=+557.326735064" Jul 21 12:36:50.693435 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:50.693375 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-tczgm" podStartSLOduration=1.746287391 podStartE2EDuration="3.693358208s" totalImagesPullingTime="1.947070817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:36:50.691226803 +0000 UTC m=+557.341640166" watchObservedRunningTime="2026-07-21 12:36:50.693358208 +0000 UTC m=+557.343771568" Jul 21 12:36:55.662348 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:36:55.662309 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cf648dd67-hqdgj" Jul 21 12:37:01.663615 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:01.663577 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-tczgm" Jul 21 12:37:01.664016 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:01.663743 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-d82c7" Jul 21 12:37:33.829752 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:33.829724 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:37:33.833514 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:33.833491 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:37:33.836052 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:33.836026 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:37:33.839225 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:33.839207 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:37:42.994151 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:42.994105 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv"] Jul 21 12:37:42.997538 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:42.997520 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.000417 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.000382 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 12:37:43.001507 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.001487 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 12:37:43.001665 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.001643 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7sbsq\"" Jul 21 12:37:43.001731 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.001683 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 21 12:37:43.009236 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.009212 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv"] Jul 21 12:37:43.072234 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.072200 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.072411 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.072258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.072411 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.072351 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc82p\" (UniqueName: \"kubernetes.io/projected/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-kube-api-access-pc82p\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.072411 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.072395 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.072603 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.072488 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.072603 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.072522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.173572 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.173536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc82p\" (UniqueName: \"kubernetes.io/projected/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-kube-api-access-pc82p\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.173572 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.173574 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.173820 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.173624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.173820 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.173644 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.173820 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.173668 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.173820 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.173698 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.174087 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.174067 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.174146 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.174102 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.174234 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.174211 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.176155 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.176132 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.176261 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.176248 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.182009 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.181981 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc82p\" (UniqueName: \"kubernetes.io/projected/391cd2e1-763f-4b3e-ab2d-230ea901d9c5-kube-api-access-pc82p\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rn4xv\" (UID: \"391cd2e1-763f-4b3e-ab2d-230ea901d9c5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.308863 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.308809 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:43.451196 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.451173 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv"] Jul 21 12:37:43.454350 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:37:43.454312 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391cd2e1_763f_4b3e_ab2d_230ea901d9c5.slice/crio-ea54e7dd52e6114351c90c2c4663bee8548cf7082766c9b54dc66401a818d94c WatchSource:0}: Error finding container ea54e7dd52e6114351c90c2c4663bee8548cf7082766c9b54dc66401a818d94c: Status 404 returned error can't find the container with id ea54e7dd52e6114351c90c2c4663bee8548cf7082766c9b54dc66401a818d94c Jul 21 12:37:43.937180 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:43.937145 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" event={"ID":"391cd2e1-763f-4b3e-ab2d-230ea901d9c5","Type":"ContainerStarted","Data":"ea54e7dd52e6114351c90c2c4663bee8548cf7082766c9b54dc66401a818d94c"} Jul 21 12:37:46.954011 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:46.953970 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" event={"ID":"391cd2e1-763f-4b3e-ab2d-230ea901d9c5","Type":"ContainerStarted","Data":"c0ec211ff60bff4f7d5f931aff73a7e5e9f1e47adfb0e0e880275a174ad22a69"} Jul 21 12:37:46.976860 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:46.976800 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" podStartSLOduration=2.128649636 podStartE2EDuration="4.976782834s" totalImagesPullingTime="2.848133198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:37:46.975991563 +0000 UTC m=+613.626404924" watchObservedRunningTime="2026-07-21 12:37:46.976782834 +0000 UTC m=+613.627196210" Jul 21 12:37:53.309015 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:53.308974 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:53.309413 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:53.309037 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:53.321854 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:53.321820 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:37:54.003067 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:37:54.003034 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rn4xv" Jul 21 12:38:14.283522 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.283488 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh"] Jul 21 12:38:14.289391 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.289370 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.292124 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.292099 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 21 12:38:14.296364 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.296336 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh"] Jul 21 12:38:14.357372 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.357332 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f95n4\" (UniqueName: \"kubernetes.io/projected/a2c20a86-3a9e-461c-b51a-900d63e1ab97-kube-api-access-f95n4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.357372 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.357375 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c20a86-3a9e-461c-b51a-900d63e1ab97-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.357581 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.357416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c20a86-3a9e-461c-b51a-900d63e1ab97-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.357581 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.357451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c20a86-3a9e-461c-b51a-900d63e1ab97-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.357581 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.357472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c20a86-3a9e-461c-b51a-900d63e1ab97-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.357581 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.357540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c20a86-3a9e-461c-b51a-900d63e1ab97-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.458162 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.458128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c20a86-3a9e-461c-b51a-900d63e1ab97-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.458364 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.458173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c20a86-3a9e-461c-b51a-900d63e1ab97-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.458364 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.458209 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c20a86-3a9e-461c-b51a-900d63e1ab97-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.458364 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.458291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f95n4\" (UniqueName: \"kubernetes.io/projected/a2c20a86-3a9e-461c-b51a-900d63e1ab97-kube-api-access-f95n4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.458364 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.458331 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c20a86-3a9e-461c-b51a-900d63e1ab97-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.458583 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.458379 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c20a86-3a9e-461c-b51a-900d63e1ab97-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.458720 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.458697 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c20a86-3a9e-461c-b51a-900d63e1ab97-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.458780 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.458737 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c20a86-3a9e-461c-b51a-900d63e1ab97-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.458780 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.458739 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c20a86-3a9e-461c-b51a-900d63e1ab97-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.460605 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.460586 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c20a86-3a9e-461c-b51a-900d63e1ab97-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.460811 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.460791 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c20a86-3a9e-461c-b51a-900d63e1ab97-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.467114 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.467086 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f95n4\" (UniqueName: \"kubernetes.io/projected/a2c20a86-3a9e-461c-b51a-900d63e1ab97-kube-api-access-f95n4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh\" (UID: \"a2c20a86-3a9e-461c-b51a-900d63e1ab97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.601276 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.601194 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:14.740552 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.740523 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh"] Jul 21 12:38:14.742680 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:38:14.742650 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c20a86_3a9e_461c_b51a_900d63e1ab97.slice/crio-ebf6a9462db4f72653c6d28c8c8f9548b50dc35c79a85fd5d88d8016a288f3c3 WatchSource:0}: Error finding container ebf6a9462db4f72653c6d28c8c8f9548b50dc35c79a85fd5d88d8016a288f3c3: Status 404 returned error can't find the container with id ebf6a9462db4f72653c6d28c8c8f9548b50dc35c79a85fd5d88d8016a288f3c3 Jul 21 12:38:14.744011 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:14.743996 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 12:38:15.105927 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:15.105892 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" event={"ID":"a2c20a86-3a9e-461c-b51a-900d63e1ab97","Type":"ContainerStarted","Data":"8abe3b0be99eee47c9d9a60b7e5f8c23e07bca1ca5d1a6465067e81a87de362a"} Jul 21 12:38:15.106118 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:15.105971 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" event={"ID":"a2c20a86-3a9e-461c-b51a-900d63e1ab97","Type":"ContainerStarted","Data":"ebf6a9462db4f72653c6d28c8c8f9548b50dc35c79a85fd5d88d8016a288f3c3"} Jul 21 12:38:15.124233 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:15.124153 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" podStartSLOduration=0.946718614 podStartE2EDuration="1.124135456s" totalImagesPullingTime="177.416842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:38:15.122841129 +0000 UTC m=+641.773254489" watchObservedRunningTime="2026-07-21 12:38:15.124135456 +0000 UTC m=+641.774548816" Jul 21 12:38:24.602173 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:24.602134 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:24.602778 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:24.602207 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:24.614805 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:24.614777 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:38:25.169548 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:38:25.169509 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh" Jul 21 12:40:45.661549 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:45.661516 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-jkqjx"] Jul 21 12:40:45.662008 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:45.661708 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" podUID="54d35ea9-b8ea-49bc-b497-8eaec0d39a0a" containerName="manager" containerID="cri-o://40f56b106a3ae8fd798498edb5aac8cea9e3490ca4f33646996134ea4427c5fe" gracePeriod=30 Jul 21 12:40:45.891217 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:45.891190 2574 generic.go:356] "Generic (PLEG): container finished" podID="54d35ea9-b8ea-49bc-b497-8eaec0d39a0a" containerID="40f56b106a3ae8fd798498edb5aac8cea9e3490ca4f33646996134ea4427c5fe" exitCode=0 Jul 21 12:40:45.891325 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:45.891263 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" event={"ID":"54d35ea9-b8ea-49bc-b497-8eaec0d39a0a","Type":"ContainerDied","Data":"40f56b106a3ae8fd798498edb5aac8cea9e3490ca4f33646996134ea4427c5fe"} Jul 21 12:40:45.910757 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:45.910730 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" Jul 21 12:40:46.060391 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:46.060362 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg4mh\" (UniqueName: \"kubernetes.io/projected/54d35ea9-b8ea-49bc-b497-8eaec0d39a0a-kube-api-access-lg4mh\") pod \"54d35ea9-b8ea-49bc-b497-8eaec0d39a0a\" (UID: \"54d35ea9-b8ea-49bc-b497-8eaec0d39a0a\") " Jul 21 12:40:46.060596 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:46.060457 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54d35ea9-b8ea-49bc-b497-8eaec0d39a0a-webhook-cert\") pod \"54d35ea9-b8ea-49bc-b497-8eaec0d39a0a\" (UID: \"54d35ea9-b8ea-49bc-b497-8eaec0d39a0a\") " Jul 21 12:40:46.062512 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:46.062479 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d35ea9-b8ea-49bc-b497-8eaec0d39a0a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "54d35ea9-b8ea-49bc-b497-8eaec0d39a0a" (UID: "54d35ea9-b8ea-49bc-b497-8eaec0d39a0a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:40:46.062512 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:46.062498 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d35ea9-b8ea-49bc-b497-8eaec0d39a0a-kube-api-access-lg4mh" (OuterVolumeSpecName: "kube-api-access-lg4mh") pod "54d35ea9-b8ea-49bc-b497-8eaec0d39a0a" (UID: "54d35ea9-b8ea-49bc-b497-8eaec0d39a0a"). InnerVolumeSpecName "kube-api-access-lg4mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:40:46.161293 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:46.161251 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg4mh\" (UniqueName: \"kubernetes.io/projected/54d35ea9-b8ea-49bc-b497-8eaec0d39a0a-kube-api-access-lg4mh\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:40:46.161293 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:46.161286 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54d35ea9-b8ea-49bc-b497-8eaec0d39a0a-webhook-cert\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:40:46.897719 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:46.897688 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" Jul 21 12:40:46.898210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:46.897740 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68fb8f6cb6-jkqjx" event={"ID":"54d35ea9-b8ea-49bc-b497-8eaec0d39a0a","Type":"ContainerDied","Data":"72546cf206cbf80b08016faf5e162ec932f11ca3151003b974e563c0bd270703"} Jul 21 12:40:46.898210 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:46.897804 2574 scope.go:117] "RemoveContainer" containerID="40f56b106a3ae8fd798498edb5aac8cea9e3490ca4f33646996134ea4427c5fe" Jul 21 12:40:46.920748 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:46.920723 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-jkqjx"] Jul 21 12:40:46.928125 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:46.928102 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-jkqjx"] Jul 21 12:40:47.923910 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:40:47.923877 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54d35ea9-b8ea-49bc-b497-8eaec0d39a0a" path="/var/lib/kubelet/pods/54d35ea9-b8ea-49bc-b497-8eaec0d39a0a/volumes" Jul 21 12:41:47.808888 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:47.808855 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-l4jx7"] Jul 21 12:41:47.809341 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:47.809240 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d35ea9-b8ea-49bc-b497-8eaec0d39a0a" containerName="manager" Jul 21 12:41:47.809341 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:47.809253 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d35ea9-b8ea-49bc-b497-8eaec0d39a0a" containerName="manager" Jul 21 12:41:47.809341 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:47.809327 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="54d35ea9-b8ea-49bc-b497-8eaec0d39a0a" containerName="manager" Jul 21 12:41:47.813364 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:47.813346 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" Jul 21 12:41:47.816196 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:47.816175 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6jfm5\"" Jul 21 12:41:47.816424 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:47.816410 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 12:41:47.817585 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:47.817570 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 12:41:47.822871 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:47.822846 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-l4jx7"] Jul 21 12:41:47.904750 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:47.904699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5bw7\" (UniqueName: \"kubernetes.io/projected/a5b81988-6b5c-4a23-9c6f-371e75214713-kube-api-access-s5bw7\") pod \"maas-controller-68fb8f6cb6-l4jx7\" (UID: \"a5b81988-6b5c-4a23-9c6f-371e75214713\") " pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" Jul 21 12:41:47.904919 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:47.904761 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5b81988-6b5c-4a23-9c6f-371e75214713-webhook-cert\") pod \"maas-controller-68fb8f6cb6-l4jx7\" (UID: \"a5b81988-6b5c-4a23-9c6f-371e75214713\") " pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" Jul 21 12:41:48.005755 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:48.005710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5bw7\" (UniqueName: \"kubernetes.io/projected/a5b81988-6b5c-4a23-9c6f-371e75214713-kube-api-access-s5bw7\") pod \"maas-controller-68fb8f6cb6-l4jx7\" (UID: \"a5b81988-6b5c-4a23-9c6f-371e75214713\") " pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" Jul 21 12:41:48.005968 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:48.005780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5b81988-6b5c-4a23-9c6f-371e75214713-webhook-cert\") pod \"maas-controller-68fb8f6cb6-l4jx7\" (UID: \"a5b81988-6b5c-4a23-9c6f-371e75214713\") " pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" Jul 21 12:41:48.008177 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:48.008159 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5b81988-6b5c-4a23-9c6f-371e75214713-webhook-cert\") pod \"maas-controller-68fb8f6cb6-l4jx7\" (UID: \"a5b81988-6b5c-4a23-9c6f-371e75214713\") " pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" Jul 21 12:41:48.017355 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:48.017331 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5bw7\" (UniqueName: \"kubernetes.io/projected/a5b81988-6b5c-4a23-9c6f-371e75214713-kube-api-access-s5bw7\") pod \"maas-controller-68fb8f6cb6-l4jx7\" (UID: \"a5b81988-6b5c-4a23-9c6f-371e75214713\") " pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" Jul 21 12:41:48.124909 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:48.124814 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" Jul 21 12:41:48.279237 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:48.279157 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-l4jx7"] Jul 21 12:41:48.281388 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:41:48.281364 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b81988_6b5c_4a23_9c6f_371e75214713.slice/crio-373313b75a6613e6e0b5d10cb7ade120929df65334ba3b54c12aed8a90c09962 WatchSource:0}: Error finding container 373313b75a6613e6e0b5d10cb7ade120929df65334ba3b54c12aed8a90c09962: Status 404 returned error can't find the container with id 373313b75a6613e6e0b5d10cb7ade120929df65334ba3b54c12aed8a90c09962 Jul 21 12:41:49.218624 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:49.218587 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" event={"ID":"a5b81988-6b5c-4a23-9c6f-371e75214713","Type":"ContainerStarted","Data":"96d123e97f524aacdfdb7db323088ff978d302de56b3bd163fac9b15bad488cd"} Jul 21 12:41:49.219075 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:49.218641 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" event={"ID":"a5b81988-6b5c-4a23-9c6f-371e75214713","Type":"ContainerStarted","Data":"373313b75a6613e6e0b5d10cb7ade120929df65334ba3b54c12aed8a90c09962"} Jul 21 12:41:49.219075 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:49.218661 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" Jul 21 12:41:49.236958 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:41:49.236877 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" podStartSLOduration=1.785847656 podStartE2EDuration="2.236860254s" totalImagesPullingTime="451.012598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:41:49.236373171 +0000 UTC m=+855.886786533" watchObservedRunningTime="2026-07-21 12:41:49.236860254 +0000 UTC m=+855.887273615" Jul 21 12:42:00.226756 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:00.226722 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" Jul 21 12:42:04.935550 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:04.935496 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-l4jx7"] Jul 21 12:42:04.936040 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:04.935723 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" podUID="a5b81988-6b5c-4a23-9c6f-371e75214713" containerName="manager" containerID="cri-o://96d123e97f524aacdfdb7db323088ff978d302de56b3bd163fac9b15bad488cd" gracePeriod=30 Jul 21 12:42:05.186430 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.186370 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" Jul 21 12:42:05.270150 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.270123 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5bw7\" (UniqueName: \"kubernetes.io/projected/a5b81988-6b5c-4a23-9c6f-371e75214713-kube-api-access-s5bw7\") pod \"a5b81988-6b5c-4a23-9c6f-371e75214713\" (UID: \"a5b81988-6b5c-4a23-9c6f-371e75214713\") " Jul 21 12:42:05.270315 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.270294 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5b81988-6b5c-4a23-9c6f-371e75214713-webhook-cert\") pod \"a5b81988-6b5c-4a23-9c6f-371e75214713\" (UID: \"a5b81988-6b5c-4a23-9c6f-371e75214713\") " Jul 21 12:42:05.272108 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.272081 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b81988-6b5c-4a23-9c6f-371e75214713-kube-api-access-s5bw7" (OuterVolumeSpecName: "kube-api-access-s5bw7") pod "a5b81988-6b5c-4a23-9c6f-371e75214713" (UID: "a5b81988-6b5c-4a23-9c6f-371e75214713"). InnerVolumeSpecName "kube-api-access-s5bw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:42:05.272223 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.272112 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b81988-6b5c-4a23-9c6f-371e75214713-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a5b81988-6b5c-4a23-9c6f-371e75214713" (UID: "a5b81988-6b5c-4a23-9c6f-371e75214713"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:42:05.311467 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.311436 2574 generic.go:356] "Generic (PLEG): container finished" podID="a5b81988-6b5c-4a23-9c6f-371e75214713" containerID="96d123e97f524aacdfdb7db323088ff978d302de56b3bd163fac9b15bad488cd" exitCode=0 Jul 21 12:42:05.311610 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.311478 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" Jul 21 12:42:05.311610 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.311521 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" event={"ID":"a5b81988-6b5c-4a23-9c6f-371e75214713","Type":"ContainerDied","Data":"96d123e97f524aacdfdb7db323088ff978d302de56b3bd163fac9b15bad488cd"} Jul 21 12:42:05.311610 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.311571 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68fb8f6cb6-l4jx7" event={"ID":"a5b81988-6b5c-4a23-9c6f-371e75214713","Type":"ContainerDied","Data":"373313b75a6613e6e0b5d10cb7ade120929df65334ba3b54c12aed8a90c09962"} Jul 21 12:42:05.311610 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.311592 2574 scope.go:117] "RemoveContainer" containerID="96d123e97f524aacdfdb7db323088ff978d302de56b3bd163fac9b15bad488cd" Jul 21 12:42:05.320974 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.320957 2574 scope.go:117] "RemoveContainer" containerID="96d123e97f524aacdfdb7db323088ff978d302de56b3bd163fac9b15bad488cd" Jul 21 12:42:05.321222 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:42:05.321201 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d123e97f524aacdfdb7db323088ff978d302de56b3bd163fac9b15bad488cd\": container with ID starting with 96d123e97f524aacdfdb7db323088ff978d302de56b3bd163fac9b15bad488cd not found: ID does not exist" containerID="96d123e97f524aacdfdb7db323088ff978d302de56b3bd163fac9b15bad488cd" Jul 21 12:42:05.321286 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.321230 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d123e97f524aacdfdb7db323088ff978d302de56b3bd163fac9b15bad488cd"} err="failed to get container status \"96d123e97f524aacdfdb7db323088ff978d302de56b3bd163fac9b15bad488cd\": rpc error: code = NotFound desc = could not find container \"96d123e97f524aacdfdb7db323088ff978d302de56b3bd163fac9b15bad488cd\": container with ID starting with 96d123e97f524aacdfdb7db323088ff978d302de56b3bd163fac9b15bad488cd not found: ID does not exist" Jul 21 12:42:05.332742 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.332721 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-l4jx7"] Jul 21 12:42:05.336704 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.336683 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-l4jx7"] Jul 21 12:42:05.371251 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.371226 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5b81988-6b5c-4a23-9c6f-371e75214713-webhook-cert\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:42:05.371251 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.371248 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5bw7\" (UniqueName: \"kubernetes.io/projected/a5b81988-6b5c-4a23-9c6f-371e75214713-kube-api-access-s5bw7\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:42:05.924484 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:05.924443 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5b81988-6b5c-4a23-9c6f-371e75214713" path="/var/lib/kubelet/pods/a5b81988-6b5c-4a23-9c6f-371e75214713/volumes" Jul 21 12:42:06.167244 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.167217 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-w9hbr"] Jul 21 12:42:06.167597 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.167588 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b81988-6b5c-4a23-9c6f-371e75214713" containerName="manager" Jul 21 12:42:06.167640 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.167599 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b81988-6b5c-4a23-9c6f-371e75214713" containerName="manager" Jul 21 12:42:06.167691 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.167681 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5b81988-6b5c-4a23-9c6f-371e75214713" containerName="manager" Jul 21 12:42:06.172187 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.172164 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68fb8f6cb6-w9hbr" Jul 21 12:42:06.176045 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.175985 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 12:42:06.176045 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.175996 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6jfm5\"" Jul 21 12:42:06.176045 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.176030 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 12:42:06.181439 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.181409 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-w9hbr"] Jul 21 12:42:06.280360 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.280317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrr4\" (UniqueName: \"kubernetes.io/projected/771bcead-5e3c-432f-b402-84affce9c8b1-kube-api-access-rwrr4\") pod \"maas-controller-68fb8f6cb6-w9hbr\" (UID: \"771bcead-5e3c-432f-b402-84affce9c8b1\") " pod="opendatahub/maas-controller-68fb8f6cb6-w9hbr" Jul 21 12:42:06.280535 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.280397 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/771bcead-5e3c-432f-b402-84affce9c8b1-webhook-cert\") pod \"maas-controller-68fb8f6cb6-w9hbr\" (UID: \"771bcead-5e3c-432f-b402-84affce9c8b1\") " pod="opendatahub/maas-controller-68fb8f6cb6-w9hbr" Jul 21 12:42:06.381107 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.381073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrr4\" (UniqueName: \"kubernetes.io/projected/771bcead-5e3c-432f-b402-84affce9c8b1-kube-api-access-rwrr4\") pod \"maas-controller-68fb8f6cb6-w9hbr\" (UID: \"771bcead-5e3c-432f-b402-84affce9c8b1\") " pod="opendatahub/maas-controller-68fb8f6cb6-w9hbr" Jul 21 12:42:06.381283 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.381135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/771bcead-5e3c-432f-b402-84affce9c8b1-webhook-cert\") pod \"maas-controller-68fb8f6cb6-w9hbr\" (UID: \"771bcead-5e3c-432f-b402-84affce9c8b1\") " pod="opendatahub/maas-controller-68fb8f6cb6-w9hbr" Jul 21 12:42:06.383618 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.383587 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/771bcead-5e3c-432f-b402-84affce9c8b1-webhook-cert\") pod \"maas-controller-68fb8f6cb6-w9hbr\" (UID: \"771bcead-5e3c-432f-b402-84affce9c8b1\") " pod="opendatahub/maas-controller-68fb8f6cb6-w9hbr" Jul 21 12:42:06.390771 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.390741 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrr4\" (UniqueName: \"kubernetes.io/projected/771bcead-5e3c-432f-b402-84affce9c8b1-kube-api-access-rwrr4\") pod \"maas-controller-68fb8f6cb6-w9hbr\" (UID: \"771bcead-5e3c-432f-b402-84affce9c8b1\") " pod="opendatahub/maas-controller-68fb8f6cb6-w9hbr" Jul 21 12:42:06.484732 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.484641 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68fb8f6cb6-w9hbr" Jul 21 12:42:06.618667 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:06.618634 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68fb8f6cb6-w9hbr"] Jul 21 12:42:07.325406 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:07.325370 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68fb8f6cb6-w9hbr" event={"ID":"771bcead-5e3c-432f-b402-84affce9c8b1","Type":"ContainerStarted","Data":"e0a4b973afd927b99d8c066a21a478b8fbcdcae6e7f01df0d0c7313474906fae"} Jul 21 12:42:07.325771 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:07.325421 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68fb8f6cb6-w9hbr" event={"ID":"771bcead-5e3c-432f-b402-84affce9c8b1","Type":"ContainerStarted","Data":"90af72445fa95c4a15fc008769bff94651f4b79c06f0c2eaf78b8023760039f4"} Jul 21 12:42:07.325771 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:07.325451 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68fb8f6cb6-w9hbr" Jul 21 12:42:07.348744 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:07.348622 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68fb8f6cb6-w9hbr" podStartSLOduration=0.882527794 podStartE2EDuration="1.348600466s" totalImagesPullingTime="466.072672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:42:07.344708135 +0000 UTC m=+873.995121497" watchObservedRunningTime="2026-07-21 12:42:07.348600466 +0000 UTC m=+873.999013831" Jul 21 12:42:18.333624 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:18.333594 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68fb8f6cb6-w9hbr" Jul 21 12:42:33.868090 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:33.868003 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:42:33.874307 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:33.874283 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:42:33.874452 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:33.874375 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:42:33.880595 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:42:33.880575 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:45:00.190853 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:00.190813 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743965-fzt6n"] Jul 21 12:45:00.196815 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:00.196782 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743965-fzt6n" Jul 21 12:45:00.201286 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:00.201260 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jq8px\"" Jul 21 12:45:00.207080 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:00.207062 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743965-fzt6n" Jul 21 12:45:00.272289 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:00.272251 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743965-fzt6n"] Jul 21 12:45:00.541448 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:00.541395 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743965-fzt6n"] Jul 21 12:45:00.544268 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:45:00.544233 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3525e52d_141a_4c8c_af85_edc4be0d8bc5.slice/crio-f33ca2b2538f4c472f0769603f11fff961ff06044a3a0b7fb5f1245ff778f439 WatchSource:0}: Error finding container f33ca2b2538f4c472f0769603f11fff961ff06044a3a0b7fb5f1245ff778f439: Status 404 returned error can't find the container with id f33ca2b2538f4c472f0769603f11fff961ff06044a3a0b7fb5f1245ff778f439 Jul 21 12:45:00.546593 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:00.546573 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 12:45:01.237899 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:01.237860 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743965-fzt6n" event={"ID":"3525e52d-141a-4c8c-af85-edc4be0d8bc5","Type":"ContainerStarted","Data":"f33ca2b2538f4c472f0769603f11fff961ff06044a3a0b7fb5f1245ff778f439"} Jul 21 12:45:06.268653 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:06.268619 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743965-fzt6n" event={"ID":"3525e52d-141a-4c8c-af85-edc4be0d8bc5","Type":"ContainerStarted","Data":"da0cccb5062230f9b7496685112d36c1da136098d81d8f3ed25ecae132fe978e"} Jul 21 12:45:07.274746 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:07.274711 2574 generic.go:356] "Generic (PLEG): container finished" podID="3525e52d-141a-4c8c-af85-edc4be0d8bc5" containerID="da0cccb5062230f9b7496685112d36c1da136098d81d8f3ed25ecae132fe978e" exitCode=0 Jul 21 12:45:07.275154 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:07.274752 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743965-fzt6n" event={"ID":"3525e52d-141a-4c8c-af85-edc4be0d8bc5","Type":"ContainerDied","Data":"da0cccb5062230f9b7496685112d36c1da136098d81d8f3ed25ecae132fe978e"} Jul 21 12:45:07.410599 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:07.410574 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743965-fzt6n" Jul 21 12:45:08.287227 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:08.287195 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743965-fzt6n" Jul 21 12:45:08.287824 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:08.287229 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743965-fzt6n" event={"ID":"3525e52d-141a-4c8c-af85-edc4be0d8bc5","Type":"ContainerDied","Data":"f33ca2b2538f4c472f0769603f11fff961ff06044a3a0b7fb5f1245ff778f439"} Jul 21 12:45:08.287824 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:45:08.287301 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f33ca2b2538f4c472f0769603f11fff961ff06044a3a0b7fb5f1245ff778f439" Jul 21 12:47:33.904247 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:47:33.904219 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:47:33.910244 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:47:33.910219 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:47:33.912645 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:47:33.912625 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:47:33.918109 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:47:33.918087 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:52:33.940298 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:52:33.940268 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:52:33.945885 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:52:33.945864 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:52:33.957870 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:52:33.957843 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:52:33.963163 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:52:33.963140 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:54:34.299779 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.299705 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k"] Jul 21 12:54:34.302135 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.300150 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3525e52d-141a-4c8c-af85-edc4be0d8bc5" containerName="cleanup" Jul 21 12:54:34.302135 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.300162 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3525e52d-141a-4c8c-af85-edc4be0d8bc5" containerName="cleanup" Jul 21 12:54:34.302135 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.300213 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3525e52d-141a-4c8c-af85-edc4be0d8bc5" containerName="cleanup" Jul 21 12:54:34.303130 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.303115 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:54:34.305804 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.305785 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9e430475-serving-cert\"" Jul 21 12:54:34.314032 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.314007 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k"] Jul 21 12:54:34.396611 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.396573 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-maas-api-tls\") pod \"maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k\" (UID: \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:54:34.396778 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.396624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmdr\" (UniqueName: \"kubernetes.io/projected/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-kube-api-access-5tmdr\") pod \"maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k\" (UID: \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:54:34.396778 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.396703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-openshift-service-ca\") pod \"maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k\" (UID: \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:54:34.497973 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.497922 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-maas-api-tls\") pod \"maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k\" (UID: \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:54:34.498175 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.497994 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmdr\" (UniqueName: \"kubernetes.io/projected/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-kube-api-access-5tmdr\") pod \"maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k\" (UID: \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:54:34.498175 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.498037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-openshift-service-ca\") pod \"maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k\" (UID: \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:54:34.498733 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.498710 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-openshift-service-ca\") pod \"maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k\" (UID: \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:54:34.500301 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.500281 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-maas-api-tls\") pod \"maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k\" (UID: \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:54:34.506700 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.506673 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmdr\" (UniqueName: \"kubernetes.io/projected/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-kube-api-access-5tmdr\") pod \"maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k\" (UID: \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:54:34.614994 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.614886 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:54:34.764701 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.764675 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k"] Jul 21 12:54:34.766156 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:54:34.766132 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a7ce4e_3f85_499f_a708_882d3a72ed2b.slice/crio-35a17795ca4f96455d73c16709fc2e4d32b115529324b06c385f3b88ddc9961a WatchSource:0}: Error finding container 35a17795ca4f96455d73c16709fc2e4d32b115529324b06c385f3b88ddc9961a: Status 404 returned error can't find the container with id 35a17795ca4f96455d73c16709fc2e4d32b115529324b06c385f3b88ddc9961a Jul 21 12:54:34.767539 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:34.767519 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 12:54:35.259336 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:35.259296 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" event={"ID":"b0a7ce4e-3f85-499f-a708-882d3a72ed2b","Type":"ContainerStarted","Data":"35a17795ca4f96455d73c16709fc2e4d32b115529324b06c385f3b88ddc9961a"} Jul 21 12:54:36.266287 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:36.266240 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" event={"ID":"b0a7ce4e-3f85-499f-a708-882d3a72ed2b","Type":"ContainerStarted","Data":"a483e1e8453d230f441aa1603be03253a80e00346b1797f8d33fcde6958ce46e"} Jul 21 12:54:36.266672 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:36.266406 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:54:36.287720 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:36.287668 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" podStartSLOduration=1.604599432 podStartE2EDuration="2.287653897s" totalImagesPullingTime="683.054465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:54:36.285562109 +0000 UTC m=+1622.935975469" watchObservedRunningTime="2026-07-21 12:54:36.287653897 +0000 UTC m=+1622.938067309" Jul 21 12:54:42.277242 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:42.277214 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:54:59.901030 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:59.900995 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2"] Jul 21 12:54:59.905317 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:59.905292 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2" Jul 21 12:54:59.908863 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:59.908842 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jq8px\"" Jul 21 12:54:59.914107 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:54:59.913870 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2"] Jul 21 12:55:00.036617 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:00.036583 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9704c2ca-5630-462a-8518-0f2e6b8e4aeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e430475-5hrd2\" (UID: \"9704c2ca-5630-462a-8518-0f2e6b8e4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2" Jul 21 12:55:00.137351 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:00.137321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9704c2ca-5630-462a-8518-0f2e6b8e4aeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e430475-5hrd2\" (UID: \"9704c2ca-5630-462a-8518-0f2e6b8e4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2" Jul 21 12:55:00.138042 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:00.138024 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9704c2ca-5630-462a-8518-0f2e6b8e4aeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e430475-5hrd2\" (UID: \"9704c2ca-5630-462a-8518-0f2e6b8e4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2" Jul 21 12:55:00.219755 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:00.219667 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2" Jul 21 12:55:00.357186 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:00.357155 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2"] Jul 21 12:55:00.360986 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:55:00.360954 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9704c2ca_5630_462a_8518_0f2e6b8e4aeb.slice/crio-e1d1805ed35bee045ed217c63f3b4b6c0993fdd77ed6df7a9c6dc5013c033253 WatchSource:0}: Error finding container e1d1805ed35bee045ed217c63f3b4b6c0993fdd77ed6df7a9c6dc5013c033253: Status 404 returned error can't find the container with id e1d1805ed35bee045ed217c63f3b4b6c0993fdd77ed6df7a9c6dc5013c033253 Jul 21 12:55:00.394199 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:00.394168 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2" event={"ID":"9704c2ca-5630-462a-8518-0f2e6b8e4aeb","Type":"ContainerStarted","Data":"e1d1805ed35bee045ed217c63f3b4b6c0993fdd77ed6df7a9c6dc5013c033253"} Jul 21 12:55:01.402103 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:01.402009 2574 generic.go:356] "Generic (PLEG): container finished" podID="9704c2ca-5630-462a-8518-0f2e6b8e4aeb" containerID="b70cccdb884a6c437b123d6b8fab1b90cc0a73cfb06e945aa5e9b217e681994f" exitCode=0 Jul 21 12:55:01.402103 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:01.402079 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2" event={"ID":"9704c2ca-5630-462a-8518-0f2e6b8e4aeb","Type":"ContainerDied","Data":"b70cccdb884a6c437b123d6b8fab1b90cc0a73cfb06e945aa5e9b217e681994f"} Jul 21 12:55:02.544527 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:02.544501 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2" Jul 21 12:55:02.659675 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:02.659566 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9704c2ca-5630-462a-8518-0f2e6b8e4aeb-maas-api-service-ca\") pod \"9704c2ca-5630-462a-8518-0f2e6b8e4aeb\" (UID: \"9704c2ca-5630-462a-8518-0f2e6b8e4aeb\") " Jul 21 12:55:02.659891 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:02.659861 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9704c2ca-5630-462a-8518-0f2e6b8e4aeb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9704c2ca-5630-462a-8518-0f2e6b8e4aeb" (UID: "9704c2ca-5630-462a-8518-0f2e6b8e4aeb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:55:02.760695 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:02.760647 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9704c2ca-5630-462a-8518-0f2e6b8e4aeb-maas-api-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:55:03.414398 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:03.414373 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2" Jul 21 12:55:03.414563 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:03.414368 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2" event={"ID":"9704c2ca-5630-462a-8518-0f2e6b8e4aeb","Type":"ContainerDied","Data":"e1d1805ed35bee045ed217c63f3b4b6c0993fdd77ed6df7a9c6dc5013c033253"} Jul 21 12:55:03.414563 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:03.414500 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1d1805ed35bee045ed217c63f3b4b6c0993fdd77ed6df7a9c6dc5013c033253" Jul 21 12:55:10.438590 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:10.438558 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k"] Jul 21 12:55:10.439161 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:10.438770 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" podUID="b0a7ce4e-3f85-499f-a708-882d3a72ed2b" containerName="maas-api" containerID="cri-o://a483e1e8453d230f441aa1603be03253a80e00346b1797f8d33fcde6958ce46e" gracePeriod=30 Jul 21 12:55:10.694465 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:10.694406 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:55:10.729971 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:10.729925 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-openshift-service-ca\") pod \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\" (UID: \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\") " Jul 21 12:55:10.730205 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:10.729979 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tmdr\" (UniqueName: \"kubernetes.io/projected/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-kube-api-access-5tmdr\") pod \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\" (UID: \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\") " Jul 21 12:55:10.730205 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:10.730093 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-maas-api-tls\") pod \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\" (UID: \"b0a7ce4e-3f85-499f-a708-882d3a72ed2b\") " Jul 21 12:55:10.730341 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:10.730314 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0a7ce4e-3f85-499f-a708-882d3a72ed2b" (UID: "b0a7ce4e-3f85-499f-a708-882d3a72ed2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:55:10.732186 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:10.732155 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0a7ce4e-3f85-499f-a708-882d3a72ed2b" (UID: "b0a7ce4e-3f85-499f-a708-882d3a72ed2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:55:10.732311 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:10.732201 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-kube-api-access-5tmdr" (OuterVolumeSpecName: "kube-api-access-5tmdr") pod "b0a7ce4e-3f85-499f-a708-882d3a72ed2b" (UID: "b0a7ce4e-3f85-499f-a708-882d3a72ed2b"). InnerVolumeSpecName "kube-api-access-5tmdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:55:10.831072 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:10.831042 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-openshift-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:55:10.831072 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:10.831069 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmdr\" (UniqueName: \"kubernetes.io/projected/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-kube-api-access-5tmdr\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:55:10.831259 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:10.831079 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a7ce4e-3f85-499f-a708-882d3a72ed2b-maas-api-tls\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:55:11.459248 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:11.459211 2574 generic.go:356] "Generic (PLEG): container finished" podID="b0a7ce4e-3f85-499f-a708-882d3a72ed2b" containerID="a483e1e8453d230f441aa1603be03253a80e00346b1797f8d33fcde6958ce46e" exitCode=0 Jul 21 12:55:11.459638 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:11.459257 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" Jul 21 12:55:11.459638 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:11.459291 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" event={"ID":"b0a7ce4e-3f85-499f-a708-882d3a72ed2b","Type":"ContainerDied","Data":"a483e1e8453d230f441aa1603be03253a80e00346b1797f8d33fcde6958ce46e"} Jul 21 12:55:11.459638 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:11.459348 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k" event={"ID":"b0a7ce4e-3f85-499f-a708-882d3a72ed2b","Type":"ContainerDied","Data":"35a17795ca4f96455d73c16709fc2e4d32b115529324b06c385f3b88ddc9961a"} Jul 21 12:55:11.459638 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:11.459371 2574 scope.go:117] "RemoveContainer" containerID="a483e1e8453d230f441aa1603be03253a80e00346b1797f8d33fcde6958ce46e" Jul 21 12:55:11.469166 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:11.469147 2574 scope.go:117] "RemoveContainer" containerID="a483e1e8453d230f441aa1603be03253a80e00346b1797f8d33fcde6958ce46e" Jul 21 12:55:11.469405 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:55:11.469388 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a483e1e8453d230f441aa1603be03253a80e00346b1797f8d33fcde6958ce46e\": container with ID starting with a483e1e8453d230f441aa1603be03253a80e00346b1797f8d33fcde6958ce46e not found: ID does not exist" containerID="a483e1e8453d230f441aa1603be03253a80e00346b1797f8d33fcde6958ce46e" Jul 21 12:55:11.469477 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:11.469411 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a483e1e8453d230f441aa1603be03253a80e00346b1797f8d33fcde6958ce46e"} err="failed to get container status \"a483e1e8453d230f441aa1603be03253a80e00346b1797f8d33fcde6958ce46e\": rpc error: code = NotFound desc = could not find container \"a483e1e8453d230f441aa1603be03253a80e00346b1797f8d33fcde6958ce46e\": container with ID starting with a483e1e8453d230f441aa1603be03253a80e00346b1797f8d33fcde6958ce46e not found: ID does not exist" Jul 21 12:55:11.492583 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:11.492558 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k"] Jul 21 12:55:11.495913 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:11.495887 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e430475-76c8c5cddc-vd74k"] Jul 21 12:55:11.924372 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:11.924339 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0a7ce4e-3f85-499f-a708-882d3a72ed2b" path="/var/lib/kubelet/pods/b0a7ce4e-3f85-499f-a708-882d3a72ed2b/volumes" Jul 21 12:55:15.001096 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:15.001051 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2"] Jul 21 12:55:15.003240 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:15.003219 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e430475-5hrd2"] Jul 21 12:55:15.925077 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:15.925047 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9704c2ca-5630-462a-8518-0f2e6b8e4aeb" path="/var/lib/kubelet/pods/9704c2ca-5630-462a-8518-0f2e6b8e4aeb/volumes" Jul 21 12:55:18.713974 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.713922 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx"] Jul 21 12:55:18.714508 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.714486 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a7ce4e-3f85-499f-a708-882d3a72ed2b" containerName="maas-api" Jul 21 12:55:18.714597 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.714511 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a7ce4e-3f85-499f-a708-882d3a72ed2b" containerName="maas-api" Jul 21 12:55:18.714597 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.714548 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9704c2ca-5630-462a-8518-0f2e6b8e4aeb" containerName="revoke-keys" Jul 21 12:55:18.714597 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.714556 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9704c2ca-5630-462a-8518-0f2e6b8e4aeb" containerName="revoke-keys" Jul 21 12:55:18.714761 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.714653 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9704c2ca-5630-462a-8518-0f2e6b8e4aeb" containerName="revoke-keys" Jul 21 12:55:18.714761 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.714664 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a7ce4e-3f85-499f-a708-882d3a72ed2b" containerName="maas-api" Jul 21 12:55:18.725395 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.725366 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:18.728328 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.728307 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-cf78342b-serving-cert\"" Jul 21 12:55:18.736249 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.736222 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx"] Jul 21 12:55:18.810297 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.810262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416da2bb-215b-461f-ba59-f3a7e0a50c60-maas-api-tls\") pod \"maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx\" (UID: \"416da2bb-215b-461f-ba59-f3a7e0a50c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:18.810490 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.810308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpgc9\" (UniqueName: \"kubernetes.io/projected/416da2bb-215b-461f-ba59-f3a7e0a50c60-kube-api-access-lpgc9\") pod \"maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx\" (UID: \"416da2bb-215b-461f-ba59-f3a7e0a50c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:18.810490 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.810404 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416da2bb-215b-461f-ba59-f3a7e0a50c60-openshift-service-ca\") pod \"maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx\" (UID: \"416da2bb-215b-461f-ba59-f3a7e0a50c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:18.911195 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.911154 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416da2bb-215b-461f-ba59-f3a7e0a50c60-maas-api-tls\") pod \"maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx\" (UID: \"416da2bb-215b-461f-ba59-f3a7e0a50c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:18.911389 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.911204 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpgc9\" (UniqueName: \"kubernetes.io/projected/416da2bb-215b-461f-ba59-f3a7e0a50c60-kube-api-access-lpgc9\") pod \"maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx\" (UID: \"416da2bb-215b-461f-ba59-f3a7e0a50c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:18.911389 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.911250 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416da2bb-215b-461f-ba59-f3a7e0a50c60-openshift-service-ca\") pod \"maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx\" (UID: \"416da2bb-215b-461f-ba59-f3a7e0a50c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:18.911913 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.911887 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416da2bb-215b-461f-ba59-f3a7e0a50c60-openshift-service-ca\") pod \"maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx\" (UID: \"416da2bb-215b-461f-ba59-f3a7e0a50c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:18.913596 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.913572 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416da2bb-215b-461f-ba59-f3a7e0a50c60-maas-api-tls\") pod \"maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx\" (UID: \"416da2bb-215b-461f-ba59-f3a7e0a50c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:18.919991 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:18.919965 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpgc9\" (UniqueName: \"kubernetes.io/projected/416da2bb-215b-461f-ba59-f3a7e0a50c60-kube-api-access-lpgc9\") pod \"maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx\" (UID: \"416da2bb-215b-461f-ba59-f3a7e0a50c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:19.037226 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:19.037194 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:19.179505 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:19.179477 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx"] Jul 21 12:55:19.181583 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:55:19.181551 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416da2bb_215b_461f_ba59_f3a7e0a50c60.slice/crio-afdc7c7c087f10aeff8af57d4c9b4a1d86da4d85ddd56bf916b5a56e77afcb22 WatchSource:0}: Error finding container afdc7c7c087f10aeff8af57d4c9b4a1d86da4d85ddd56bf916b5a56e77afcb22: Status 404 returned error can't find the container with id afdc7c7c087f10aeff8af57d4c9b4a1d86da4d85ddd56bf916b5a56e77afcb22 Jul 21 12:55:19.503007 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:19.502976 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" event={"ID":"416da2bb-215b-461f-ba59-f3a7e0a50c60","Type":"ContainerStarted","Data":"afdc7c7c087f10aeff8af57d4c9b4a1d86da4d85ddd56bf916b5a56e77afcb22"} Jul 21 12:55:20.510107 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:20.510065 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" event={"ID":"416da2bb-215b-461f-ba59-f3a7e0a50c60","Type":"ContainerStarted","Data":"50457bb9c5049d419503a8920f45275302b3d215f976b0762aa62abbd36c1f42"} Jul 21 12:55:20.510539 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:20.510133 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:20.529708 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:20.529658 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" podStartSLOduration=1.795055018 podStartE2EDuration="2.529643544s" totalImagesPullingTime="734.588526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:55:20.528695773 +0000 UTC m=+1667.179109134" watchObservedRunningTime="2026-07-21 12:55:20.529643544 +0000 UTC m=+1667.180056904" Jul 21 12:55:26.521559 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:26.521516 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:43.663468 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:43.663427 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf"] Jul 21 12:55:43.669474 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:43.669442 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf" Jul 21 12:55:43.673665 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:43.673620 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jq8px\"" Jul 21 12:55:43.678340 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:43.678312 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf"] Jul 21 12:55:43.828763 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:43.828729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b623ae62-7a26-4ca5-a376-93b3dd3e5df8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf\" (UID: \"b623ae62-7a26-4ca5-a376-93b3dd3e5df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf" Jul 21 12:55:43.929470 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:43.929375 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b623ae62-7a26-4ca5-a376-93b3dd3e5df8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf\" (UID: \"b623ae62-7a26-4ca5-a376-93b3dd3e5df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf" Jul 21 12:55:43.929998 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:43.929975 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b623ae62-7a26-4ca5-a376-93b3dd3e5df8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf\" (UID: \"b623ae62-7a26-4ca5-a376-93b3dd3e5df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf" Jul 21 12:55:43.984381 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:43.984347 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf" Jul 21 12:55:44.130094 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:44.130063 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf"] Jul 21 12:55:44.131907 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:55:44.131881 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb623ae62_7a26_4ca5_a376_93b3dd3e5df8.slice/crio-f0f8d0e9d4c7432658ef4451c67efe2d1f1a51e86f94a84b5a1483d710370c93 WatchSource:0}: Error finding container f0f8d0e9d4c7432658ef4451c67efe2d1f1a51e86f94a84b5a1483d710370c93: Status 404 returned error can't find the container with id f0f8d0e9d4c7432658ef4451c67efe2d1f1a51e86f94a84b5a1483d710370c93 Jul 21 12:55:44.641616 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:44.641514 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf" event={"ID":"b623ae62-7a26-4ca5-a376-93b3dd3e5df8","Type":"ContainerStarted","Data":"b5c29122bf9a1e7e05f51c5edbe97c2226837d683ce88addfb59fec43a6a4be0"} Jul 21 12:55:44.641616 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:44.641583 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf" event={"ID":"b623ae62-7a26-4ca5-a376-93b3dd3e5df8","Type":"ContainerStarted","Data":"f0f8d0e9d4c7432658ef4451c67efe2d1f1a51e86f94a84b5a1483d710370c93"} Jul 21 12:55:44.658324 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:44.658194 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf" podStartSLOduration=1.658177448 podStartE2EDuration="1.658177448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:55:44.657745228 +0000 UTC m=+1691.308158585" watchObservedRunningTime="2026-07-21 12:55:44.658177448 +0000 UTC m=+1691.308590810" Jul 21 12:55:45.648920 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:45.648880 2574 generic.go:356] "Generic (PLEG): container finished" podID="b623ae62-7a26-4ca5-a376-93b3dd3e5df8" containerID="b5c29122bf9a1e7e05f51c5edbe97c2226837d683ce88addfb59fec43a6a4be0" exitCode=0 Jul 21 12:55:45.649314 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:45.648929 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf" event={"ID":"b623ae62-7a26-4ca5-a376-93b3dd3e5df8","Type":"ContainerDied","Data":"b5c29122bf9a1e7e05f51c5edbe97c2226837d683ce88addfb59fec43a6a4be0"} Jul 21 12:55:46.794337 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:46.794311 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf" Jul 21 12:55:46.958586 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:46.958500 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b623ae62-7a26-4ca5-a376-93b3dd3e5df8-maas-api-service-ca\") pod \"b623ae62-7a26-4ca5-a376-93b3dd3e5df8\" (UID: \"b623ae62-7a26-4ca5-a376-93b3dd3e5df8\") " Jul 21 12:55:46.958795 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:46.958773 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b623ae62-7a26-4ca5-a376-93b3dd3e5df8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b623ae62-7a26-4ca5-a376-93b3dd3e5df8" (UID: "b623ae62-7a26-4ca5-a376-93b3dd3e5df8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:55:47.060167 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:47.060135 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b623ae62-7a26-4ca5-a376-93b3dd3e5df8-maas-api-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:55:47.659857 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:47.659817 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf" event={"ID":"b623ae62-7a26-4ca5-a376-93b3dd3e5df8","Type":"ContainerDied","Data":"f0f8d0e9d4c7432658ef4451c67efe2d1f1a51e86f94a84b5a1483d710370c93"} Jul 21 12:55:47.660046 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:47.659868 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0f8d0e9d4c7432658ef4451c67efe2d1f1a51e86f94a84b5a1483d710370c93" Jul 21 12:55:47.660046 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:47.659829 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf" Jul 21 12:55:54.165389 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.165356 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx"] Jul 21 12:55:54.165838 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.165738 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" podUID="416da2bb-215b-461f-ba59-f3a7e0a50c60" containerName="maas-api" containerID="cri-o://50457bb9c5049d419503a8920f45275302b3d215f976b0762aa62abbd36c1f42" gracePeriod=30 Jul 21 12:55:54.424863 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.424791 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:54.427874 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.427851 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpgc9\" (UniqueName: \"kubernetes.io/projected/416da2bb-215b-461f-ba59-f3a7e0a50c60-kube-api-access-lpgc9\") pod \"416da2bb-215b-461f-ba59-f3a7e0a50c60\" (UID: \"416da2bb-215b-461f-ba59-f3a7e0a50c60\") " Jul 21 12:55:54.428016 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.427913 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416da2bb-215b-461f-ba59-f3a7e0a50c60-openshift-service-ca\") pod \"416da2bb-215b-461f-ba59-f3a7e0a50c60\" (UID: \"416da2bb-215b-461f-ba59-f3a7e0a50c60\") " Jul 21 12:55:54.428016 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.427992 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416da2bb-215b-461f-ba59-f3a7e0a50c60-maas-api-tls\") pod \"416da2bb-215b-461f-ba59-f3a7e0a50c60\" (UID: \"416da2bb-215b-461f-ba59-f3a7e0a50c60\") " Jul 21 12:55:54.428268 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.428231 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416da2bb-215b-461f-ba59-f3a7e0a50c60-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "416da2bb-215b-461f-ba59-f3a7e0a50c60" (UID: "416da2bb-215b-461f-ba59-f3a7e0a50c60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:55:54.429991 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.429963 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416da2bb-215b-461f-ba59-f3a7e0a50c60-kube-api-access-lpgc9" (OuterVolumeSpecName: "kube-api-access-lpgc9") pod "416da2bb-215b-461f-ba59-f3a7e0a50c60" (UID: "416da2bb-215b-461f-ba59-f3a7e0a50c60"). InnerVolumeSpecName "kube-api-access-lpgc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:55:54.430200 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.430179 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416da2bb-215b-461f-ba59-f3a7e0a50c60-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "416da2bb-215b-461f-ba59-f3a7e0a50c60" (UID: "416da2bb-215b-461f-ba59-f3a7e0a50c60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:55:54.528870 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.528826 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416da2bb-215b-461f-ba59-f3a7e0a50c60-openshift-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:55:54.528870 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.528864 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416da2bb-215b-461f-ba59-f3a7e0a50c60-maas-api-tls\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:55:54.528870 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.528877 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpgc9\" (UniqueName: \"kubernetes.io/projected/416da2bb-215b-461f-ba59-f3a7e0a50c60-kube-api-access-lpgc9\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:55:54.698389 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.698296 2574 generic.go:356] "Generic (PLEG): container finished" podID="416da2bb-215b-461f-ba59-f3a7e0a50c60" containerID="50457bb9c5049d419503a8920f45275302b3d215f976b0762aa62abbd36c1f42" exitCode=0 Jul 21 12:55:54.698389 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.698345 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" Jul 21 12:55:54.698389 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.698352 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" event={"ID":"416da2bb-215b-461f-ba59-f3a7e0a50c60","Type":"ContainerDied","Data":"50457bb9c5049d419503a8920f45275302b3d215f976b0762aa62abbd36c1f42"} Jul 21 12:55:54.698645 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.698404 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx" event={"ID":"416da2bb-215b-461f-ba59-f3a7e0a50c60","Type":"ContainerDied","Data":"afdc7c7c087f10aeff8af57d4c9b4a1d86da4d85ddd56bf916b5a56e77afcb22"} Jul 21 12:55:54.698645 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.698436 2574 scope.go:117] "RemoveContainer" containerID="50457bb9c5049d419503a8920f45275302b3d215f976b0762aa62abbd36c1f42" Jul 21 12:55:54.708527 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.708506 2574 scope.go:117] "RemoveContainer" containerID="50457bb9c5049d419503a8920f45275302b3d215f976b0762aa62abbd36c1f42" Jul 21 12:55:54.708776 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:55:54.708752 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50457bb9c5049d419503a8920f45275302b3d215f976b0762aa62abbd36c1f42\": container with ID starting with 50457bb9c5049d419503a8920f45275302b3d215f976b0762aa62abbd36c1f42 not found: ID does not exist" containerID="50457bb9c5049d419503a8920f45275302b3d215f976b0762aa62abbd36c1f42" Jul 21 12:55:54.708845 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.708789 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50457bb9c5049d419503a8920f45275302b3d215f976b0762aa62abbd36c1f42"} err="failed to get container status \"50457bb9c5049d419503a8920f45275302b3d215f976b0762aa62abbd36c1f42\": rpc error: code = NotFound desc = could not find container \"50457bb9c5049d419503a8920f45275302b3d215f976b0762aa62abbd36c1f42\": container with ID starting with 50457bb9c5049d419503a8920f45275302b3d215f976b0762aa62abbd36c1f42 not found: ID does not exist" Jul 21 12:55:54.721777 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.721754 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx"] Jul 21 12:55:54.726307 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:54.726277 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf78342b-86984f6cf4-zgnjx"] Jul 21 12:55:55.924567 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:55.924490 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="416da2bb-215b-461f-ba59-f3a7e0a50c60" path="/var/lib/kubelet/pods/416da2bb-215b-461f-ba59-f3a7e0a50c60/volumes" Jul 21 12:55:58.773166 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:58.773129 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf"] Jul 21 12:55:58.778966 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:58.778926 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf78342b-ffqcf"] Jul 21 12:55:59.924357 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:55:59.924325 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b623ae62-7a26-4ca5-a376-93b3dd3e5df8" path="/var/lib/kubelet/pods/b623ae62-7a26-4ca5-a376-93b3dd3e5df8/volumes" Jul 21 12:56:02.961323 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:02.961290 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm"] Jul 21 12:56:02.961702 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:02.961689 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="416da2bb-215b-461f-ba59-f3a7e0a50c60" containerName="maas-api" Jul 21 12:56:02.961702 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:02.961703 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="416da2bb-215b-461f-ba59-f3a7e0a50c60" containerName="maas-api" Jul 21 12:56:02.961782 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:02.961710 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b623ae62-7a26-4ca5-a376-93b3dd3e5df8" containerName="revoke-keys" Jul 21 12:56:02.961782 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:02.961716 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b623ae62-7a26-4ca5-a376-93b3dd3e5df8" containerName="revoke-keys" Jul 21 12:56:02.961782 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:02.961781 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="416da2bb-215b-461f-ba59-f3a7e0a50c60" containerName="maas-api" Jul 21 12:56:02.961887 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:02.961791 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b623ae62-7a26-4ca5-a376-93b3dd3e5df8" containerName="revoke-keys" Jul 21 12:56:02.964033 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:02.964000 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:02.968836 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:02.968811 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0645c77a-serving-cert\"" Jul 21 12:56:02.981709 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:02.981688 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm"] Jul 21 12:56:03.000012 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:02.999982 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6hq9\" (UniqueName: \"kubernetes.io/projected/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-kube-api-access-f6hq9\") pod \"maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm\" (UID: \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:03.000133 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:03.000023 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-openshift-service-ca\") pod \"maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm\" (UID: \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:03.000133 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:03.000122 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-maas-api-tls\") pod \"maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm\" (UID: \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:03.101293 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:03.101255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-maas-api-tls\") pod \"maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm\" (UID: \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:03.101474 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:56:03.101407 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-serving-cert: secret "maas-api-e2e-ait-0645c77a-serving-cert" not found Jul 21 12:56:03.101474 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:03.101418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6hq9\" (UniqueName: \"kubernetes.io/projected/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-kube-api-access-f6hq9\") pod \"maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm\" (UID: \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:03.101557 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:56:03.101483 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-maas-api-tls podName:69d5d5e1-2d5d-4e68-b35a-e23675bbb978 nodeName:}" failed. No retries permitted until 2026-07-21 12:56:03.601460265 +0000 UTC m=+1710.251873607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-maas-api-tls") pod "maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" (UID: "69d5d5e1-2d5d-4e68-b35a-e23675bbb978") : secret "maas-api-e2e-ait-0645c77a-serving-cert" not found Jul 21 12:56:03.101557 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:03.101525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-openshift-service-ca\") pod \"maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm\" (UID: \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:03.102257 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:03.102238 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-openshift-service-ca\") pod \"maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm\" (UID: \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:03.113681 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:03.113656 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6hq9\" (UniqueName: \"kubernetes.io/projected/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-kube-api-access-f6hq9\") pod \"maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm\" (UID: \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:03.606387 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:03.606348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-maas-api-tls\") pod \"maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm\" (UID: \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:03.606566 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:56:03.606474 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-serving-cert: secret "maas-api-e2e-ait-0645c77a-serving-cert" not found Jul 21 12:56:03.606566 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:56:03.606536 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-maas-api-tls podName:69d5d5e1-2d5d-4e68-b35a-e23675bbb978 nodeName:}" failed. No retries permitted until 2026-07-21 12:56:04.606521682 +0000 UTC m=+1711.256935026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-maas-api-tls") pod "maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" (UID: "69d5d5e1-2d5d-4e68-b35a-e23675bbb978") : secret "maas-api-e2e-ait-0645c77a-serving-cert" not found Jul 21 12:56:04.615653 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:04.615620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-maas-api-tls\") pod \"maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm\" (UID: \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:04.618218 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:04.618184 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-maas-api-tls\") pod \"maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm\" (UID: \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:04.774605 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:04.774570 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:04.914360 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:04.914323 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm"] Jul 21 12:56:04.918861 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:56:04.918242 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d5d5e1_2d5d_4e68_b35a_e23675bbb978.slice/crio-4df4111ea803f9ab2fc8be7ae9a35ca66cd41faf29eb00ba2873deb882171e3d WatchSource:0}: Error finding container 4df4111ea803f9ab2fc8be7ae9a35ca66cd41faf29eb00ba2873deb882171e3d: Status 404 returned error can't find the container with id 4df4111ea803f9ab2fc8be7ae9a35ca66cd41faf29eb00ba2873deb882171e3d Jul 21 12:56:05.765083 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:05.765048 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" event={"ID":"69d5d5e1-2d5d-4e68-b35a-e23675bbb978","Type":"ContainerStarted","Data":"4ff537608dca5ae4a45370f8529e986658ba58f83e8795fd40a632bc387cf978"} Jul 21 12:56:05.765505 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:05.765098 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" event={"ID":"69d5d5e1-2d5d-4e68-b35a-e23675bbb978","Type":"ContainerStarted","Data":"4df4111ea803f9ab2fc8be7ae9a35ca66cd41faf29eb00ba2873deb882171e3d"} Jul 21 12:56:05.765505 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:05.765154 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:05.784898 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:05.784847 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" podStartSLOduration=3.2745531359999998 podStartE2EDuration="3.784832339s" totalImagesPullingTime="510.279203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:56:05.782079265 +0000 UTC m=+1712.432492627" watchObservedRunningTime="2026-07-21 12:56:05.784832339 +0000 UTC m=+1712.435245699" Jul 21 12:56:11.776677 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:11.776649 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:28.571658 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:28.571618 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln"] Jul 21 12:56:28.575646 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:28.575627 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln" Jul 21 12:56:28.578685 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:28.578664 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jq8px\"" Jul 21 12:56:28.584297 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:28.584271 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln"] Jul 21 12:56:28.747003 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:28.746967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c18e183-93c2-41f1-9f5b-dc94569bda28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0645c77a-dnhln\" (UID: \"6c18e183-93c2-41f1-9f5b-dc94569bda28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln" Jul 21 12:56:28.848514 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:28.848416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c18e183-93c2-41f1-9f5b-dc94569bda28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0645c77a-dnhln\" (UID: \"6c18e183-93c2-41f1-9f5b-dc94569bda28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln" Jul 21 12:56:28.849117 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:28.849096 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c18e183-93c2-41f1-9f5b-dc94569bda28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0645c77a-dnhln\" (UID: \"6c18e183-93c2-41f1-9f5b-dc94569bda28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln" Jul 21 12:56:28.888268 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:28.888236 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln" Jul 21 12:56:29.033962 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:29.033922 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln"] Jul 21 12:56:29.035223 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:56:29.035194 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c18e183_93c2_41f1_9f5b_dc94569bda28.slice/crio-73ba1b707643862a0a73012404808959ce9e981834ee74bb62dc5148e9cc8374 WatchSource:0}: Error finding container 73ba1b707643862a0a73012404808959ce9e981834ee74bb62dc5148e9cc8374: Status 404 returned error can't find the container with id 73ba1b707643862a0a73012404808959ce9e981834ee74bb62dc5148e9cc8374 Jul 21 12:56:29.901530 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:29.901494 2574 generic.go:356] "Generic (PLEG): container finished" podID="6c18e183-93c2-41f1-9f5b-dc94569bda28" containerID="e61b058398072d494d3ebadc77335a04eff7e0bf549ed1169bbb55f0c15c7c89" exitCode=0 Jul 21 12:56:29.902011 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:29.901591 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln" event={"ID":"6c18e183-93c2-41f1-9f5b-dc94569bda28","Type":"ContainerDied","Data":"e61b058398072d494d3ebadc77335a04eff7e0bf549ed1169bbb55f0c15c7c89"} Jul 21 12:56:29.902011 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:29.901643 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln" event={"ID":"6c18e183-93c2-41f1-9f5b-dc94569bda28","Type":"ContainerStarted","Data":"73ba1b707643862a0a73012404808959ce9e981834ee74bb62dc5148e9cc8374"} Jul 21 12:56:31.038148 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:31.038122 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln" Jul 21 12:56:31.172001 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:31.171875 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c18e183-93c2-41f1-9f5b-dc94569bda28-maas-api-service-ca\") pod \"6c18e183-93c2-41f1-9f5b-dc94569bda28\" (UID: \"6c18e183-93c2-41f1-9f5b-dc94569bda28\") " Jul 21 12:56:31.172319 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:31.172295 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c18e183-93c2-41f1-9f5b-dc94569bda28-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c18e183-93c2-41f1-9f5b-dc94569bda28" (UID: "6c18e183-93c2-41f1-9f5b-dc94569bda28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:56:31.273081 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:31.273045 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c18e183-93c2-41f1-9f5b-dc94569bda28-maas-api-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:56:31.913740 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:31.913702 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln" event={"ID":"6c18e183-93c2-41f1-9f5b-dc94569bda28","Type":"ContainerDied","Data":"73ba1b707643862a0a73012404808959ce9e981834ee74bb62dc5148e9cc8374"} Jul 21 12:56:31.913969 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:31.913791 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73ba1b707643862a0a73012404808959ce9e981834ee74bb62dc5148e9cc8374" Jul 21 12:56:31.913969 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:31.913716 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln" Jul 21 12:56:39.113087 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.113030 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm"] Jul 21 12:56:39.113543 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.113319 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" podUID="69d5d5e1-2d5d-4e68-b35a-e23675bbb978" containerName="maas-api" containerID="cri-o://4ff537608dca5ae4a45370f8529e986658ba58f83e8795fd40a632bc387cf978" gracePeriod=30 Jul 21 12:56:39.375594 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.375539 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:39.439571 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.439538 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6hq9\" (UniqueName: \"kubernetes.io/projected/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-kube-api-access-f6hq9\") pod \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\" (UID: \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\") " Jul 21 12:56:39.439764 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.439590 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-openshift-service-ca\") pod \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\" (UID: \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\") " Jul 21 12:56:39.439764 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.439630 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-maas-api-tls\") pod \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\" (UID: \"69d5d5e1-2d5d-4e68-b35a-e23675bbb978\") " Jul 21 12:56:39.439985 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.439932 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "69d5d5e1-2d5d-4e68-b35a-e23675bbb978" (UID: "69d5d5e1-2d5d-4e68-b35a-e23675bbb978"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:56:39.441731 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.441706 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-kube-api-access-f6hq9" (OuterVolumeSpecName: "kube-api-access-f6hq9") pod "69d5d5e1-2d5d-4e68-b35a-e23675bbb978" (UID: "69d5d5e1-2d5d-4e68-b35a-e23675bbb978"). InnerVolumeSpecName "kube-api-access-f6hq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:56:39.441822 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.441772 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "69d5d5e1-2d5d-4e68-b35a-e23675bbb978" (UID: "69d5d5e1-2d5d-4e68-b35a-e23675bbb978"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:56:39.541017 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.540978 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6hq9\" (UniqueName: \"kubernetes.io/projected/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-kube-api-access-f6hq9\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:56:39.541017 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.541008 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-openshift-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:56:39.541017 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.541018 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d5d5e1-2d5d-4e68-b35a-e23675bbb978-maas-api-tls\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:56:39.959384 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.959350 2574 generic.go:356] "Generic (PLEG): container finished" podID="69d5d5e1-2d5d-4e68-b35a-e23675bbb978" containerID="4ff537608dca5ae4a45370f8529e986658ba58f83e8795fd40a632bc387cf978" exitCode=0 Jul 21 12:56:39.959563 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.959390 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" Jul 21 12:56:39.959563 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.959423 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" event={"ID":"69d5d5e1-2d5d-4e68-b35a-e23675bbb978","Type":"ContainerDied","Data":"4ff537608dca5ae4a45370f8529e986658ba58f83e8795fd40a632bc387cf978"} Jul 21 12:56:39.959563 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.959474 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm" event={"ID":"69d5d5e1-2d5d-4e68-b35a-e23675bbb978","Type":"ContainerDied","Data":"4df4111ea803f9ab2fc8be7ae9a35ca66cd41faf29eb00ba2873deb882171e3d"} Jul 21 12:56:39.959563 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.959496 2574 scope.go:117] "RemoveContainer" containerID="4ff537608dca5ae4a45370f8529e986658ba58f83e8795fd40a632bc387cf978" Jul 21 12:56:39.969043 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.968985 2574 scope.go:117] "RemoveContainer" containerID="4ff537608dca5ae4a45370f8529e986658ba58f83e8795fd40a632bc387cf978" Jul 21 12:56:39.969249 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:56:39.969229 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff537608dca5ae4a45370f8529e986658ba58f83e8795fd40a632bc387cf978\": container with ID starting with 4ff537608dca5ae4a45370f8529e986658ba58f83e8795fd40a632bc387cf978 not found: ID does not exist" containerID="4ff537608dca5ae4a45370f8529e986658ba58f83e8795fd40a632bc387cf978" Jul 21 12:56:39.969298 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.969258 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff537608dca5ae4a45370f8529e986658ba58f83e8795fd40a632bc387cf978"} err="failed to get container status \"4ff537608dca5ae4a45370f8529e986658ba58f83e8795fd40a632bc387cf978\": rpc error: code = NotFound desc = could not find container \"4ff537608dca5ae4a45370f8529e986658ba58f83e8795fd40a632bc387cf978\": container with ID starting with 4ff537608dca5ae4a45370f8529e986658ba58f83e8795fd40a632bc387cf978 not found: ID does not exist" Jul 21 12:56:39.980088 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.980061 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm"] Jul 21 12:56:39.982320 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:39.982300 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0645c77a-58b9c9cdc9-sg8fm"] Jul 21 12:56:41.924512 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:41.924475 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69d5d5e1-2d5d-4e68-b35a-e23675bbb978" path="/var/lib/kubelet/pods/69d5d5e1-2d5d-4e68-b35a-e23675bbb978/volumes" Jul 21 12:56:43.646564 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:43.646534 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln"] Jul 21 12:56:43.648779 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:43.648754 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0645c77a-dnhln"] Jul 21 12:56:43.925645 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:43.925567 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c18e183-93c2-41f1-9f5b-dc94569bda28" path="/var/lib/kubelet/pods/6c18e183-93c2-41f1-9f5b-dc94569bda28/volumes" Jul 21 12:56:47.954271 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:47.954239 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g"] Jul 21 12:56:47.954645 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:47.954634 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d5d5e1-2d5d-4e68-b35a-e23675bbb978" containerName="maas-api" Jul 21 12:56:47.954645 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:47.954646 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d5d5e1-2d5d-4e68-b35a-e23675bbb978" containerName="maas-api" Jul 21 12:56:47.954736 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:47.954665 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c18e183-93c2-41f1-9f5b-dc94569bda28" containerName="revoke-keys" Jul 21 12:56:47.954736 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:47.954672 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c18e183-93c2-41f1-9f5b-dc94569bda28" containerName="revoke-keys" Jul 21 12:56:47.954800 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:47.954750 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d5d5e1-2d5d-4e68-b35a-e23675bbb978" containerName="maas-api" Jul 21 12:56:47.954800 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:47.954763 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c18e183-93c2-41f1-9f5b-dc94569bda28" containerName="revoke-keys" Jul 21 12:56:47.959424 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:47.959403 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:56:47.962759 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:47.962727 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6b5db8f8-serving-cert\"" Jul 21 12:56:47.970578 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:47.970540 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g"] Jul 21 12:56:48.019989 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:48.019952 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c464158b-b247-4b7f-bb7f-df18583f3044-openshift-service-ca\") pod \"maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g\" (UID: \"c464158b-b247-4b7f-bb7f-df18583f3044\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:56:48.020189 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:48.020057 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c464158b-b247-4b7f-bb7f-df18583f3044-maas-api-tls\") pod \"maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g\" (UID: \"c464158b-b247-4b7f-bb7f-df18583f3044\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:56:48.020189 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:48.020152 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqsvr\" (UniqueName: \"kubernetes.io/projected/c464158b-b247-4b7f-bb7f-df18583f3044-kube-api-access-mqsvr\") pod \"maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g\" (UID: \"c464158b-b247-4b7f-bb7f-df18583f3044\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:56:48.121330 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:48.121292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c464158b-b247-4b7f-bb7f-df18583f3044-maas-api-tls\") pod \"maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g\" (UID: \"c464158b-b247-4b7f-bb7f-df18583f3044\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:56:48.121504 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:48.121436 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqsvr\" (UniqueName: \"kubernetes.io/projected/c464158b-b247-4b7f-bb7f-df18583f3044-kube-api-access-mqsvr\") pod \"maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g\" (UID: \"c464158b-b247-4b7f-bb7f-df18583f3044\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:56:48.121504 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:56:48.121452 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-serving-cert: secret "maas-api-e2e-derive-6b5db8f8-serving-cert" not found Jul 21 12:56:48.121504 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:48.121472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c464158b-b247-4b7f-bb7f-df18583f3044-openshift-service-ca\") pod \"maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g\" (UID: \"c464158b-b247-4b7f-bb7f-df18583f3044\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:56:48.121643 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:56:48.121533 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c464158b-b247-4b7f-bb7f-df18583f3044-maas-api-tls podName:c464158b-b247-4b7f-bb7f-df18583f3044 nodeName:}" failed. No retries permitted until 2026-07-21 12:56:48.621514716 +0000 UTC m=+1755.271928055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c464158b-b247-4b7f-bb7f-df18583f3044-maas-api-tls") pod "maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" (UID: "c464158b-b247-4b7f-bb7f-df18583f3044") : secret "maas-api-e2e-derive-6b5db8f8-serving-cert" not found Jul 21 12:56:48.122101 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:48.122084 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c464158b-b247-4b7f-bb7f-df18583f3044-openshift-service-ca\") pod \"maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g\" (UID: \"c464158b-b247-4b7f-bb7f-df18583f3044\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:56:48.132324 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:48.132295 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqsvr\" (UniqueName: \"kubernetes.io/projected/c464158b-b247-4b7f-bb7f-df18583f3044-kube-api-access-mqsvr\") pod \"maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g\" (UID: \"c464158b-b247-4b7f-bb7f-df18583f3044\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:56:48.625849 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:48.625810 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c464158b-b247-4b7f-bb7f-df18583f3044-maas-api-tls\") pod \"maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g\" (UID: \"c464158b-b247-4b7f-bb7f-df18583f3044\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:56:48.628348 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:48.628326 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c464158b-b247-4b7f-bb7f-df18583f3044-maas-api-tls\") pod \"maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g\" (UID: \"c464158b-b247-4b7f-bb7f-df18583f3044\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:56:48.870808 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:48.870772 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:56:49.014777 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:49.014748 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g"] Jul 21 12:56:49.016230 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:56:49.016208 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc464158b_b247_4b7f_bb7f_df18583f3044.slice/crio-d502c3cce7d333b9cbb4cdc5d404feb32586799d4bcfe456632e17bbd417e3a0 WatchSource:0}: Error finding container d502c3cce7d333b9cbb4cdc5d404feb32586799d4bcfe456632e17bbd417e3a0: Status 404 returned error can't find the container with id d502c3cce7d333b9cbb4cdc5d404feb32586799d4bcfe456632e17bbd417e3a0 Jul 21 12:56:50.022122 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:50.022083 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" event={"ID":"c464158b-b247-4b7f-bb7f-df18583f3044","Type":"ContainerStarted","Data":"baa76929416932f1f9e5e772062fc075412ccba52d38fc021943ab53641eea97"} Jul 21 12:56:50.022528 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:50.022136 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" event={"ID":"c464158b-b247-4b7f-bb7f-df18583f3044","Type":"ContainerStarted","Data":"d502c3cce7d333b9cbb4cdc5d404feb32586799d4bcfe456632e17bbd417e3a0"} Jul 21 12:56:50.022528 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:50.022219 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:56:50.045129 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:50.045081 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" podStartSLOduration=2.680671528 podStartE2EDuration="3.045064037s" totalImagesPullingTime="364.392509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:56:50.043960469 +0000 UTC m=+1756.694373838" watchObservedRunningTime="2026-07-21 12:56:50.045064037 +0000 UTC m=+1756.695477397" Jul 21 12:56:56.033347 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:56:56.033314 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:57:12.726330 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:12.726294 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc"] Jul 21 12:57:12.729920 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:12.729902 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc" Jul 21 12:57:12.733103 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:12.733073 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jq8px\"" Jul 21 12:57:12.736842 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:12.736812 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc"] Jul 21 12:57:12.757986 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:12.757930 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaaa4a5-e07d-4366-b076-26a4f06570f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc\" (UID: \"ddaaa4a5-e07d-4366-b076-26a4f06570f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc" Jul 21 12:57:12.858546 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:12.858506 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaaa4a5-e07d-4366-b076-26a4f06570f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc\" (UID: \"ddaaa4a5-e07d-4366-b076-26a4f06570f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc" Jul 21 12:57:12.859193 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:12.859171 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaaa4a5-e07d-4366-b076-26a4f06570f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc\" (UID: \"ddaaa4a5-e07d-4366-b076-26a4f06570f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc" Jul 21 12:57:13.042854 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:13.042821 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc" Jul 21 12:57:13.182389 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:13.182360 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc"] Jul 21 12:57:13.184327 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:57:13.184295 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddaaa4a5_e07d_4366_b076_26a4f06570f7.slice/crio-fb00c6e563e6c4f9434c1003528e263acae2582d0afe022435a6874c5fa3fbba WatchSource:0}: Error finding container fb00c6e563e6c4f9434c1003528e263acae2582d0afe022435a6874c5fa3fbba: Status 404 returned error can't find the container with id fb00c6e563e6c4f9434c1003528e263acae2582d0afe022435a6874c5fa3fbba Jul 21 12:57:14.157729 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:14.157645 2574 generic.go:356] "Generic (PLEG): container finished" podID="ddaaa4a5-e07d-4366-b076-26a4f06570f7" containerID="d58daabde643092a30fbc8ba5737f818dd4230cae5f3a2e397cf4ebdca2b027f" exitCode=0 Jul 21 12:57:14.158165 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:14.157732 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc" event={"ID":"ddaaa4a5-e07d-4366-b076-26a4f06570f7","Type":"ContainerDied","Data":"d58daabde643092a30fbc8ba5737f818dd4230cae5f3a2e397cf4ebdca2b027f"} Jul 21 12:57:14.158165 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:14.157780 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc" event={"ID":"ddaaa4a5-e07d-4366-b076-26a4f06570f7","Type":"ContainerStarted","Data":"fb00c6e563e6c4f9434c1003528e263acae2582d0afe022435a6874c5fa3fbba"} Jul 21 12:57:15.298906 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:15.298879 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc" Jul 21 12:57:15.379775 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:15.379744 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaaa4a5-e07d-4366-b076-26a4f06570f7-maas-api-service-ca\") pod \"ddaaa4a5-e07d-4366-b076-26a4f06570f7\" (UID: \"ddaaa4a5-e07d-4366-b076-26a4f06570f7\") " Jul 21 12:57:15.380198 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:15.380178 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddaaa4a5-e07d-4366-b076-26a4f06570f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ddaaa4a5-e07d-4366-b076-26a4f06570f7" (UID: "ddaaa4a5-e07d-4366-b076-26a4f06570f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:57:15.481213 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:15.481124 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaaa4a5-e07d-4366-b076-26a4f06570f7-maas-api-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:57:16.171059 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:16.171031 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc" Jul 21 12:57:16.171262 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:16.171069 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc" event={"ID":"ddaaa4a5-e07d-4366-b076-26a4f06570f7","Type":"ContainerDied","Data":"fb00c6e563e6c4f9434c1003528e263acae2582d0afe022435a6874c5fa3fbba"} Jul 21 12:57:16.171262 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:16.171118 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb00c6e563e6c4f9434c1003528e263acae2582d0afe022435a6874c5fa3fbba" Jul 21 12:57:23.128819 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:23.128778 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g"] Jul 21 12:57:23.129313 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:23.129060 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" podUID="c464158b-b247-4b7f-bb7f-df18583f3044" containerName="maas-api" containerID="cri-o://baa76929416932f1f9e5e772062fc075412ccba52d38fc021943ab53641eea97" gracePeriod=30 Jul 21 12:57:23.378997 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:23.378900 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:57:23.467044 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:23.466998 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqsvr\" (UniqueName: \"kubernetes.io/projected/c464158b-b247-4b7f-bb7f-df18583f3044-kube-api-access-mqsvr\") pod \"c464158b-b247-4b7f-bb7f-df18583f3044\" (UID: \"c464158b-b247-4b7f-bb7f-df18583f3044\") " Jul 21 12:57:23.467237 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:23.467059 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c464158b-b247-4b7f-bb7f-df18583f3044-openshift-service-ca\") pod \"c464158b-b247-4b7f-bb7f-df18583f3044\" (UID: \"c464158b-b247-4b7f-bb7f-df18583f3044\") " Jul 21 12:57:23.467237 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:23.467088 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c464158b-b247-4b7f-bb7f-df18583f3044-maas-api-tls\") pod \"c464158b-b247-4b7f-bb7f-df18583f3044\" (UID: \"c464158b-b247-4b7f-bb7f-df18583f3044\") " Jul 21 12:57:23.467432 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:23.467403 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c464158b-b247-4b7f-bb7f-df18583f3044-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c464158b-b247-4b7f-bb7f-df18583f3044" (UID: "c464158b-b247-4b7f-bb7f-df18583f3044"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:57:23.469275 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:23.469243 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c464158b-b247-4b7f-bb7f-df18583f3044-kube-api-access-mqsvr" (OuterVolumeSpecName: "kube-api-access-mqsvr") pod "c464158b-b247-4b7f-bb7f-df18583f3044" (UID: "c464158b-b247-4b7f-bb7f-df18583f3044"). InnerVolumeSpecName "kube-api-access-mqsvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:57:23.469275 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:23.469264 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c464158b-b247-4b7f-bb7f-df18583f3044-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c464158b-b247-4b7f-bb7f-df18583f3044" (UID: "c464158b-b247-4b7f-bb7f-df18583f3044"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:57:23.568635 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:23.568595 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqsvr\" (UniqueName: \"kubernetes.io/projected/c464158b-b247-4b7f-bb7f-df18583f3044-kube-api-access-mqsvr\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:57:23.568635 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:23.568627 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c464158b-b247-4b7f-bb7f-df18583f3044-openshift-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:57:23.568635 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:23.568639 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c464158b-b247-4b7f-bb7f-df18583f3044-maas-api-tls\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:57:24.221690 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:24.221588 2574 generic.go:356] "Generic (PLEG): container finished" podID="c464158b-b247-4b7f-bb7f-df18583f3044" containerID="baa76929416932f1f9e5e772062fc075412ccba52d38fc021943ab53641eea97" exitCode=0 Jul 21 12:57:24.221690 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:24.221650 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" Jul 21 12:57:24.221690 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:24.221662 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" event={"ID":"c464158b-b247-4b7f-bb7f-df18583f3044","Type":"ContainerDied","Data":"baa76929416932f1f9e5e772062fc075412ccba52d38fc021943ab53641eea97"} Jul 21 12:57:24.222235 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:24.221745 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g" event={"ID":"c464158b-b247-4b7f-bb7f-df18583f3044","Type":"ContainerDied","Data":"d502c3cce7d333b9cbb4cdc5d404feb32586799d4bcfe456632e17bbd417e3a0"} Jul 21 12:57:24.222235 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:24.221772 2574 scope.go:117] "RemoveContainer" containerID="baa76929416932f1f9e5e772062fc075412ccba52d38fc021943ab53641eea97" Jul 21 12:57:24.230969 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:24.230929 2574 scope.go:117] "RemoveContainer" containerID="baa76929416932f1f9e5e772062fc075412ccba52d38fc021943ab53641eea97" Jul 21 12:57:24.231215 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:57:24.231199 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa76929416932f1f9e5e772062fc075412ccba52d38fc021943ab53641eea97\": container with ID starting with baa76929416932f1f9e5e772062fc075412ccba52d38fc021943ab53641eea97 not found: ID does not exist" containerID="baa76929416932f1f9e5e772062fc075412ccba52d38fc021943ab53641eea97" Jul 21 12:57:24.231264 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:24.231223 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa76929416932f1f9e5e772062fc075412ccba52d38fc021943ab53641eea97"} err="failed to get container status \"baa76929416932f1f9e5e772062fc075412ccba52d38fc021943ab53641eea97\": rpc error: code = NotFound desc = could not find container \"baa76929416932f1f9e5e772062fc075412ccba52d38fc021943ab53641eea97\": container with ID starting with baa76929416932f1f9e5e772062fc075412ccba52d38fc021943ab53641eea97 not found: ID does not exist" Jul 21 12:57:24.241363 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:24.241334 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g"] Jul 21 12:57:24.243835 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:24.243814 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6b5db8f8-656fd4c8d-9nf9g"] Jul 21 12:57:25.924422 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:25.924347 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c464158b-b247-4b7f-bb7f-df18583f3044" path="/var/lib/kubelet/pods/c464158b-b247-4b7f-bb7f-df18583f3044/volumes" Jul 21 12:57:27.816262 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:27.816230 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc"] Jul 21 12:57:27.821752 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:27.821725 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6b5db8f8-lmjmc"] Jul 21 12:57:27.924895 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:27.924857 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddaaa4a5-e07d-4366-b076-26a4f06570f7" path="/var/lib/kubelet/pods/ddaaa4a5-e07d-4366-b076-26a4f06570f7/volumes" Jul 21 12:57:33.986042 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:33.986013 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:57:33.991609 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:33.991587 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:57:33.999159 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:33.999134 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 12:57:34.004480 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:57:34.004461 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 12:59:13.388066 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.387982 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr"] Jul 21 12:59:13.388562 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.388388 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c464158b-b247-4b7f-bb7f-df18583f3044" containerName="maas-api" Jul 21 12:59:13.388562 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.388400 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c464158b-b247-4b7f-bb7f-df18583f3044" containerName="maas-api" Jul 21 12:59:13.388562 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.388411 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddaaa4a5-e07d-4366-b076-26a4f06570f7" containerName="revoke-keys" Jul 21 12:59:13.388562 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.388417 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddaaa4a5-e07d-4366-b076-26a4f06570f7" containerName="revoke-keys" Jul 21 12:59:13.388562 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.388479 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddaaa4a5-e07d-4366-b076-26a4f06570f7" containerName="revoke-keys" Jul 21 12:59:13.388562 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.388491 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c464158b-b247-4b7f-bb7f-df18583f3044" containerName="maas-api" Jul 21 12:59:13.391358 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.391337 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:13.397932 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.397909 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-464cd24a-serving-cert\"" Jul 21 12:59:13.403026 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.402999 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr"] Jul 21 12:59:13.489981 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.489923 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-maas-api-tls\") pod \"maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr\" (UID: \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:13.490152 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.490031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5dr\" (UniqueName: \"kubernetes.io/projected/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-kube-api-access-xg5dr\") pod \"maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr\" (UID: \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:13.490152 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.490063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-openshift-service-ca\") pod \"maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr\" (UID: \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:13.590523 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.590485 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-maas-api-tls\") pod \"maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr\" (UID: \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:13.590718 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.590562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5dr\" (UniqueName: \"kubernetes.io/projected/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-kube-api-access-xg5dr\") pod \"maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr\" (UID: \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:13.590718 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.590589 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-openshift-service-ca\") pod \"maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr\" (UID: \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:13.590718 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:59:13.590641 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-serving-cert: secret "maas-api-e2e-mt-464cd24a-serving-cert" not found Jul 21 12:59:13.590912 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:59:13.590729 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-maas-api-tls podName:64c8f7f7-253d-4c3c-b8dd-769791d03ad7 nodeName:}" failed. No retries permitted until 2026-07-21 12:59:14.090708071 +0000 UTC m=+1900.741121413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-maas-api-tls") pod "maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" (UID: "64c8f7f7-253d-4c3c-b8dd-769791d03ad7") : secret "maas-api-e2e-mt-464cd24a-serving-cert" not found Jul 21 12:59:13.591202 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.591184 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-openshift-service-ca\") pod \"maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr\" (UID: \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:13.599917 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:13.599891 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5dr\" (UniqueName: \"kubernetes.io/projected/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-kube-api-access-xg5dr\") pod \"maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr\" (UID: \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:14.095517 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:14.095481 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-maas-api-tls\") pod \"maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr\" (UID: \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:14.097963 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:14.097916 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-maas-api-tls\") pod \"maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr\" (UID: \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:14.303428 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:14.303392 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:14.662099 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:14.662072 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr"] Jul 21 12:59:14.663427 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:59:14.663399 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c8f7f7_253d_4c3c_b8dd_769791d03ad7.slice/crio-d6b9d37af009b8d467627f2e8999cafb8d236e8d46442944303f2d0a6bf1279d WatchSource:0}: Error finding container d6b9d37af009b8d467627f2e8999cafb8d236e8d46442944303f2d0a6bf1279d: Status 404 returned error can't find the container with id d6b9d37af009b8d467627f2e8999cafb8d236e8d46442944303f2d0a6bf1279d Jul 21 12:59:14.816855 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:14.816820 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" event={"ID":"64c8f7f7-253d-4c3c-b8dd-769791d03ad7","Type":"ContainerStarted","Data":"d6b9d37af009b8d467627f2e8999cafb8d236e8d46442944303f2d0a6bf1279d"} Jul 21 12:59:15.823437 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:15.823396 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" event={"ID":"64c8f7f7-253d-4c3c-b8dd-769791d03ad7","Type":"ContainerStarted","Data":"25368cb4f3d2a2eee77bca29ede1d89b6f97a501aba554e23c2e82fd7d426332"} Jul 21 12:59:15.823850 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:15.823559 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:15.843022 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:15.842901 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" podStartSLOduration=2.4366618190000002 podStartE2EDuration="2.842886989s" totalImagesPullingTime="406.22517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:59:15.841466115 +0000 UTC m=+1902.491879476" watchObservedRunningTime="2026-07-21 12:59:15.842886989 +0000 UTC m=+1902.493300349" Jul 21 12:59:21.835701 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:21.835671 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:25.034749 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:25.034705 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz"] Jul 21 12:59:25.038382 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:25.038363 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz" Jul 21 12:59:25.040962 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:25.040917 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jq8px\"" Jul 21 12:59:25.045889 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:25.045848 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz"] Jul 21 12:59:25.092041 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:25.092002 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc336e0e-ed44-462f-8bdf-0c289b9c1ba0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-464cd24a-42zhz\" (UID: \"dc336e0e-ed44-462f-8bdf-0c289b9c1ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz" Jul 21 12:59:25.193477 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:25.193441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc336e0e-ed44-462f-8bdf-0c289b9c1ba0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-464cd24a-42zhz\" (UID: \"dc336e0e-ed44-462f-8bdf-0c289b9c1ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz" Jul 21 12:59:25.194162 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:25.194138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc336e0e-ed44-462f-8bdf-0c289b9c1ba0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-464cd24a-42zhz\" (UID: \"dc336e0e-ed44-462f-8bdf-0c289b9c1ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz" Jul 21 12:59:25.350823 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:25.350743 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz" Jul 21 12:59:25.490588 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:25.490561 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz"] Jul 21 12:59:25.493139 ip-10-0-138-11 kubenswrapper[2574]: W0721 12:59:25.493106 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc336e0e_ed44_462f_8bdf_0c289b9c1ba0.slice/crio-d5e3cfc39f10d526c6496895b3903710511a6f7c7034c9fe85ec4408c8963711 WatchSource:0}: Error finding container d5e3cfc39f10d526c6496895b3903710511a6f7c7034c9fe85ec4408c8963711: Status 404 returned error can't find the container with id d5e3cfc39f10d526c6496895b3903710511a6f7c7034c9fe85ec4408c8963711 Jul 21 12:59:25.885673 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:25.885638 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz" event={"ID":"dc336e0e-ed44-462f-8bdf-0c289b9c1ba0","Type":"ContainerStarted","Data":"234ad54fb07a1b018639956b98190fd460d6215d0a95b97cf186e2b97b25b902"} Jul 21 12:59:25.885838 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:25.885704 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz" event={"ID":"dc336e0e-ed44-462f-8bdf-0c289b9c1ba0","Type":"ContainerStarted","Data":"d5e3cfc39f10d526c6496895b3903710511a6f7c7034c9fe85ec4408c8963711"} Jul 21 12:59:25.901683 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:25.901631 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz" podStartSLOduration=0.901615691 podStartE2EDuration="901.615691ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:59:25.89971317 +0000 UTC m=+1912.550126532" watchObservedRunningTime="2026-07-21 12:59:25.901615691 +0000 UTC m=+1912.552029088" Jul 21 12:59:26.892726 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:26.892694 2574 generic.go:356] "Generic (PLEG): container finished" podID="dc336e0e-ed44-462f-8bdf-0c289b9c1ba0" containerID="234ad54fb07a1b018639956b98190fd460d6215d0a95b97cf186e2b97b25b902" exitCode=0 Jul 21 12:59:26.893150 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:26.892744 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz" event={"ID":"dc336e0e-ed44-462f-8bdf-0c289b9c1ba0","Type":"ContainerDied","Data":"234ad54fb07a1b018639956b98190fd460d6215d0a95b97cf186e2b97b25b902"} Jul 21 12:59:28.024553 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:28.024523 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz" Jul 21 12:59:28.120411 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:28.120375 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc336e0e-ed44-462f-8bdf-0c289b9c1ba0-maas-api-service-ca\") pod \"dc336e0e-ed44-462f-8bdf-0c289b9c1ba0\" (UID: \"dc336e0e-ed44-462f-8bdf-0c289b9c1ba0\") " Jul 21 12:59:28.120726 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:28.120705 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc336e0e-ed44-462f-8bdf-0c289b9c1ba0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc336e0e-ed44-462f-8bdf-0c289b9c1ba0" (UID: "dc336e0e-ed44-462f-8bdf-0c289b9c1ba0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:59:28.221710 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:28.221627 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc336e0e-ed44-462f-8bdf-0c289b9c1ba0-maas-api-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:59:28.904342 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:28.904308 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz" event={"ID":"dc336e0e-ed44-462f-8bdf-0c289b9c1ba0","Type":"ContainerDied","Data":"d5e3cfc39f10d526c6496895b3903710511a6f7c7034c9fe85ec4408c8963711"} Jul 21 12:59:28.904514 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:28.904355 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5e3cfc39f10d526c6496895b3903710511a6f7c7034c9fe85ec4408c8963711" Jul 21 12:59:28.904514 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:28.904359 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz" Jul 21 12:59:35.533883 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.533842 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr"] Jul 21 12:59:35.536241 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.534085 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" podUID="64c8f7f7-253d-4c3c-b8dd-769791d03ad7" containerName="maas-api" containerID="cri-o://25368cb4f3d2a2eee77bca29ede1d89b6f97a501aba554e23c2e82fd7d426332" gracePeriod=30 Jul 21 12:59:35.785161 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.785099 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:35.886639 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.886610 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-maas-api-tls\") pod \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\" (UID: \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\") " Jul 21 12:59:35.886809 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.886655 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-openshift-service-ca\") pod \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\" (UID: \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\") " Jul 21 12:59:35.886809 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.886735 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg5dr\" (UniqueName: \"kubernetes.io/projected/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-kube-api-access-xg5dr\") pod \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\" (UID: \"64c8f7f7-253d-4c3c-b8dd-769791d03ad7\") " Jul 21 12:59:35.887088 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.887062 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "64c8f7f7-253d-4c3c-b8dd-769791d03ad7" (UID: "64c8f7f7-253d-4c3c-b8dd-769791d03ad7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:59:35.888610 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.888587 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "64c8f7f7-253d-4c3c-b8dd-769791d03ad7" (UID: "64c8f7f7-253d-4c3c-b8dd-769791d03ad7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:59:35.888730 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.888712 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-kube-api-access-xg5dr" (OuterVolumeSpecName: "kube-api-access-xg5dr") pod "64c8f7f7-253d-4c3c-b8dd-769791d03ad7" (UID: "64c8f7f7-253d-4c3c-b8dd-769791d03ad7"). InnerVolumeSpecName "kube-api-access-xg5dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:59:35.944869 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.944842 2574 generic.go:356] "Generic (PLEG): container finished" podID="64c8f7f7-253d-4c3c-b8dd-769791d03ad7" containerID="25368cb4f3d2a2eee77bca29ede1d89b6f97a501aba554e23c2e82fd7d426332" exitCode=0 Jul 21 12:59:35.945006 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.944884 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" Jul 21 12:59:35.945006 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.944904 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" event={"ID":"64c8f7f7-253d-4c3c-b8dd-769791d03ad7","Type":"ContainerDied","Data":"25368cb4f3d2a2eee77bca29ede1d89b6f97a501aba554e23c2e82fd7d426332"} Jul 21 12:59:35.945006 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.944958 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr" event={"ID":"64c8f7f7-253d-4c3c-b8dd-769791d03ad7","Type":"ContainerDied","Data":"d6b9d37af009b8d467627f2e8999cafb8d236e8d46442944303f2d0a6bf1279d"} Jul 21 12:59:35.945006 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.944980 2574 scope.go:117] "RemoveContainer" containerID="25368cb4f3d2a2eee77bca29ede1d89b6f97a501aba554e23c2e82fd7d426332" Jul 21 12:59:35.954105 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.954084 2574 scope.go:117] "RemoveContainer" containerID="25368cb4f3d2a2eee77bca29ede1d89b6f97a501aba554e23c2e82fd7d426332" Jul 21 12:59:35.954356 ip-10-0-138-11 kubenswrapper[2574]: E0721 12:59:35.954338 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25368cb4f3d2a2eee77bca29ede1d89b6f97a501aba554e23c2e82fd7d426332\": container with ID starting with 25368cb4f3d2a2eee77bca29ede1d89b6f97a501aba554e23c2e82fd7d426332 not found: ID does not exist" containerID="25368cb4f3d2a2eee77bca29ede1d89b6f97a501aba554e23c2e82fd7d426332" Jul 21 12:59:35.954405 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.954365 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25368cb4f3d2a2eee77bca29ede1d89b6f97a501aba554e23c2e82fd7d426332"} err="failed to get container status \"25368cb4f3d2a2eee77bca29ede1d89b6f97a501aba554e23c2e82fd7d426332\": rpc error: code = NotFound desc = could not find container \"25368cb4f3d2a2eee77bca29ede1d89b6f97a501aba554e23c2e82fd7d426332\": container with ID starting with 25368cb4f3d2a2eee77bca29ede1d89b6f97a501aba554e23c2e82fd7d426332 not found: ID does not exist" Jul 21 12:59:35.962626 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.962604 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr"] Jul 21 12:59:35.967286 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.967260 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464cd24a-5649449d4f-tq7lr"] Jul 21 12:59:35.987934 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.987907 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-openshift-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:59:35.987934 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.987929 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg5dr\" (UniqueName: \"kubernetes.io/projected/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-kube-api-access-xg5dr\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:59:35.988095 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:35.987955 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64c8f7f7-253d-4c3c-b8dd-769791d03ad7-maas-api-tls\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 12:59:37.924258 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:37.924229 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64c8f7f7-253d-4c3c-b8dd-769791d03ad7" path="/var/lib/kubelet/pods/64c8f7f7-253d-4c3c-b8dd-769791d03ad7/volumes" Jul 21 12:59:40.124555 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:40.124509 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz"] Jul 21 12:59:40.127559 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:40.127534 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464cd24a-42zhz"] Jul 21 12:59:41.924383 ip-10-0-138-11 kubenswrapper[2574]: I0721 12:59:41.924350 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc336e0e-ed44-462f-8bdf-0c289b9c1ba0" path="/var/lib/kubelet/pods/dc336e0e-ed44-462f-8bdf-0c289b9c1ba0/volumes" Jul 21 13:00:09.407010 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.406976 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g"] Jul 21 13:00:09.407591 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.407574 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc336e0e-ed44-462f-8bdf-0c289b9c1ba0" containerName="revoke-keys" Jul 21 13:00:09.407665 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.407594 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc336e0e-ed44-462f-8bdf-0c289b9c1ba0" containerName="revoke-keys" Jul 21 13:00:09.407665 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.407633 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c8f7f7-253d-4c3c-b8dd-769791d03ad7" containerName="maas-api" Jul 21 13:00:09.407665 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.407644 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c8f7f7-253d-4c3c-b8dd-769791d03ad7" containerName="maas-api" Jul 21 13:00:09.407817 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.407726 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc336e0e-ed44-462f-8bdf-0c289b9c1ba0" containerName="revoke-keys" Jul 21 13:00:09.407817 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.407743 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="64c8f7f7-253d-4c3c-b8dd-769791d03ad7" containerName="maas-api" Jul 21 13:00:09.410898 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.410876 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:00:09.413513 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.413489 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-dd701081-serving-cert\"" Jul 21 13:00:09.424085 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.424047 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g"] Jul 21 13:00:09.496827 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.496791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-openshift-service-ca\") pod \"maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g\" (UID: \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:00:09.497033 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.496840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-maas-api-tls\") pod \"maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g\" (UID: \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:00:09.497033 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.496903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chbsz\" (UniqueName: \"kubernetes.io/projected/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-kube-api-access-chbsz\") pod \"maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g\" (UID: \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:00:09.597484 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.597449 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-openshift-service-ca\") pod \"maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g\" (UID: \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:00:09.597661 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.597498 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-maas-api-tls\") pod \"maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g\" (UID: \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:00:09.597661 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.597617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chbsz\" (UniqueName: \"kubernetes.io/projected/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-kube-api-access-chbsz\") pod \"maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g\" (UID: \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:00:09.597753 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:00:09.597738 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-serving-cert: secret "maas-api-e2e-mt-dd701081-serving-cert" not found Jul 21 13:00:09.597811 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:00:09.597801 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-maas-api-tls podName:1283f3ad-e81e-43fe-ac59-1d3263fa58ea nodeName:}" failed. No retries permitted until 2026-07-21 13:00:10.097782912 +0000 UTC m=+1956.748196253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-maas-api-tls") pod "maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" (UID: "1283f3ad-e81e-43fe-ac59-1d3263fa58ea") : secret "maas-api-e2e-mt-dd701081-serving-cert" not found Jul 21 13:00:09.598246 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.598229 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-openshift-service-ca\") pod \"maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g\" (UID: \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:00:09.614013 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:09.613988 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chbsz\" (UniqueName: \"kubernetes.io/projected/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-kube-api-access-chbsz\") pod \"maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g\" (UID: \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:00:10.101799 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:10.101767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-maas-api-tls\") pod \"maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g\" (UID: \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:00:10.102016 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:00:10.101921 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-serving-cert: secret "maas-api-e2e-mt-dd701081-serving-cert" not found Jul 21 13:00:10.102016 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:00:10.102010 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-maas-api-tls podName:1283f3ad-e81e-43fe-ac59-1d3263fa58ea nodeName:}" failed. No retries permitted until 2026-07-21 13:00:11.10199462 +0000 UTC m=+1957.752407959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-maas-api-tls") pod "maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" (UID: "1283f3ad-e81e-43fe-ac59-1d3263fa58ea") : secret "maas-api-e2e-mt-dd701081-serving-cert" not found Jul 21 13:00:11.112008 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:11.111972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-maas-api-tls\") pod \"maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g\" (UID: \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:00:11.114322 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:11.114299 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-maas-api-tls\") pod \"maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g\" (UID: \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:00:11.223040 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:11.222996 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:00:11.585734 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:11.585702 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g"] Jul 21 13:00:11.589285 ip-10-0-138-11 kubenswrapper[2574]: W0721 13:00:11.589240 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1283f3ad_e81e_43fe_ac59_1d3263fa58ea.slice/crio-ed938c39bd34c2654147ee3e2930615d64bcbbc64a439d34a804a59d01811b8d WatchSource:0}: Error finding container ed938c39bd34c2654147ee3e2930615d64bcbbc64a439d34a804a59d01811b8d: Status 404 returned error can't find the container with id ed938c39bd34c2654147ee3e2930615d64bcbbc64a439d34a804a59d01811b8d Jul 21 13:00:11.590623 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:11.590604 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 13:00:12.145211 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:12.145172 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" event={"ID":"1283f3ad-e81e-43fe-ac59-1d3263fa58ea","Type":"ContainerStarted","Data":"ed938c39bd34c2654147ee3e2930615d64bcbbc64a439d34a804a59d01811b8d"} Jul 21 13:00:13.154105 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:13.154070 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" event={"ID":"1283f3ad-e81e-43fe-ac59-1d3263fa58ea","Type":"ContainerStarted","Data":"929411c84209c449fa0441e66d474932c038090c85b63f12dacc848efca869e2"} Jul 21 13:00:13.154480 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:13.154188 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:00:13.171731 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:13.171684 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" podStartSLOduration=2.723257997 podStartE2EDuration="4.171668493s" totalImagesPullingTime="1.448410496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:00:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:00:13.171111884 +0000 UTC m=+1959.821525248" watchObservedRunningTime="2026-07-21 13:00:13.171668493 +0000 UTC m=+1959.822081854" Jul 21 13:00:19.166166 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:00:19.166133 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:01:02.953766 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:02.953729 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd"] Jul 21 13:01:02.957441 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:02.957423 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd" Jul 21 13:01:02.960371 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:02.960348 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jq8px\"" Jul 21 13:01:02.965112 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:02.965089 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd"] Jul 21 13:01:03.091027 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:03.090991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7614a1-2bc9-458d-b596-2282d3b6d146-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd701081-dk9zd\" (UID: \"5b7614a1-2bc9-458d-b596-2282d3b6d146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd" Jul 21 13:01:03.191636 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:03.191603 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7614a1-2bc9-458d-b596-2282d3b6d146-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd701081-dk9zd\" (UID: \"5b7614a1-2bc9-458d-b596-2282d3b6d146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd" Jul 21 13:01:03.192255 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:03.192235 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7614a1-2bc9-458d-b596-2282d3b6d146-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd701081-dk9zd\" (UID: \"5b7614a1-2bc9-458d-b596-2282d3b6d146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd" Jul 21 13:01:03.269933 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:03.269904 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd" Jul 21 13:01:03.412973 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:03.412931 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd"] Jul 21 13:01:03.413867 ip-10-0-138-11 kubenswrapper[2574]: W0721 13:01:03.413833 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7614a1_2bc9_458d_b596_2282d3b6d146.slice/crio-b940523daaaba3eb855b8515a43b0ebccd6dec793a9b00cd244a46f155423f86 WatchSource:0}: Error finding container b940523daaaba3eb855b8515a43b0ebccd6dec793a9b00cd244a46f155423f86: Status 404 returned error can't find the container with id b940523daaaba3eb855b8515a43b0ebccd6dec793a9b00cd244a46f155423f86 Jul 21 13:01:03.437245 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:03.437208 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd" event={"ID":"5b7614a1-2bc9-458d-b596-2282d3b6d146","Type":"ContainerStarted","Data":"b940523daaaba3eb855b8515a43b0ebccd6dec793a9b00cd244a46f155423f86"} Jul 21 13:01:04.444726 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:04.444622 2574 generic.go:356] "Generic (PLEG): container finished" podID="5b7614a1-2bc9-458d-b596-2282d3b6d146" containerID="096d57251032660fa54f622e75c2a3da738fa3ce5b746ef75e415789db13fd5a" exitCode=0 Jul 21 13:01:04.444726 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:04.444711 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd" event={"ID":"5b7614a1-2bc9-458d-b596-2282d3b6d146","Type":"ContainerDied","Data":"096d57251032660fa54f622e75c2a3da738fa3ce5b746ef75e415789db13fd5a"} Jul 21 13:01:05.590398 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:05.590365 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd" Jul 21 13:01:05.717302 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:05.717208 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7614a1-2bc9-458d-b596-2282d3b6d146-maas-api-service-ca\") pod \"5b7614a1-2bc9-458d-b596-2282d3b6d146\" (UID: \"5b7614a1-2bc9-458d-b596-2282d3b6d146\") " Jul 21 13:01:05.717575 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:05.717549 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7614a1-2bc9-458d-b596-2282d3b6d146-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b7614a1-2bc9-458d-b596-2282d3b6d146" (UID: "5b7614a1-2bc9-458d-b596-2282d3b6d146"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:01:05.817963 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:05.817907 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7614a1-2bc9-458d-b596-2282d3b6d146-maas-api-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:01:06.457962 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:06.457918 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd" Jul 21 13:01:06.458141 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:06.457921 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd" event={"ID":"5b7614a1-2bc9-458d-b596-2282d3b6d146","Type":"ContainerDied","Data":"b940523daaaba3eb855b8515a43b0ebccd6dec793a9b00cd244a46f155423f86"} Jul 21 13:01:06.458141 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:06.458061 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b940523daaaba3eb855b8515a43b0ebccd6dec793a9b00cd244a46f155423f86" Jul 21 13:01:18.405510 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:18.405473 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g"] Jul 21 13:01:18.405993 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:18.405701 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" podUID="1283f3ad-e81e-43fe-ac59-1d3263fa58ea" containerName="maas-api" containerID="cri-o://929411c84209c449fa0441e66d474932c038090c85b63f12dacc848efca869e2" gracePeriod=30 Jul 21 13:01:18.532793 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:18.532760 2574 generic.go:356] "Generic (PLEG): container finished" podID="1283f3ad-e81e-43fe-ac59-1d3263fa58ea" containerID="929411c84209c449fa0441e66d474932c038090c85b63f12dacc848efca869e2" exitCode=0 Jul 21 13:01:18.532928 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:18.532794 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" event={"ID":"1283f3ad-e81e-43fe-ac59-1d3263fa58ea","Type":"ContainerDied","Data":"929411c84209c449fa0441e66d474932c038090c85b63f12dacc848efca869e2"} Jul 21 13:01:18.657558 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:18.657504 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:01:18.839858 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:18.839824 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-openshift-service-ca\") pod \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\" (UID: \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\") " Jul 21 13:01:18.840058 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:18.839989 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-maas-api-tls\") pod \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\" (UID: \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\") " Jul 21 13:01:18.840058 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:18.840027 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chbsz\" (UniqueName: \"kubernetes.io/projected/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-kube-api-access-chbsz\") pod \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\" (UID: \"1283f3ad-e81e-43fe-ac59-1d3263fa58ea\") " Jul 21 13:01:18.840243 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:18.840219 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1283f3ad-e81e-43fe-ac59-1d3263fa58ea" (UID: "1283f3ad-e81e-43fe-ac59-1d3263fa58ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:01:18.842146 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:18.842110 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1283f3ad-e81e-43fe-ac59-1d3263fa58ea" (UID: "1283f3ad-e81e-43fe-ac59-1d3263fa58ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:01:18.842262 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:18.842225 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-kube-api-access-chbsz" (OuterVolumeSpecName: "kube-api-access-chbsz") pod "1283f3ad-e81e-43fe-ac59-1d3263fa58ea" (UID: "1283f3ad-e81e-43fe-ac59-1d3263fa58ea"). InnerVolumeSpecName "kube-api-access-chbsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:01:18.941172 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:18.941073 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-openshift-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:01:18.941172 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:18.941114 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-maas-api-tls\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:01:18.941172 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:18.941124 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chbsz\" (UniqueName: \"kubernetes.io/projected/1283f3ad-e81e-43fe-ac59-1d3263fa58ea-kube-api-access-chbsz\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:01:19.540137 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:19.540103 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" Jul 21 13:01:19.540578 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:19.540100 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g" event={"ID":"1283f3ad-e81e-43fe-ac59-1d3263fa58ea","Type":"ContainerDied","Data":"ed938c39bd34c2654147ee3e2930615d64bcbbc64a439d34a804a59d01811b8d"} Jul 21 13:01:19.540578 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:19.540242 2574 scope.go:117] "RemoveContainer" containerID="929411c84209c449fa0441e66d474932c038090c85b63f12dacc848efca869e2" Jul 21 13:01:19.562904 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:19.562879 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g"] Jul 21 13:01:19.568879 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:19.568859 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd701081-5cdbcd7678-52s4g"] Jul 21 13:01:19.923816 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:19.923734 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1283f3ad-e81e-43fe-ac59-1d3263fa58ea" path="/var/lib/kubelet/pods/1283f3ad-e81e-43fe-ac59-1d3263fa58ea/volumes" Jul 21 13:01:23.030087 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:23.030038 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd"] Jul 21 13:01:23.032893 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:23.032868 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd701081-dk9zd"] Jul 21 13:01:23.924318 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:23.924282 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b7614a1-2bc9-458d-b596-2282d3b6d146" path="/var/lib/kubelet/pods/5b7614a1-2bc9-458d-b596-2282d3b6d146/volumes" Jul 21 13:01:34.111036 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:01:34.111003 2574 scope.go:117] "RemoveContainer" containerID="b70cccdb884a6c437b123d6b8fab1b90cc0a73cfb06e945aa5e9b217e681994f" Jul 21 13:02:34.022730 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:02:34.022703 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 13:02:34.028613 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:02:34.028587 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 13:02:34.036692 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:02:34.036669 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 13:02:34.042324 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:02:34.042303 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 13:02:34.137278 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:02:34.137244 2574 scope.go:117] "RemoveContainer" containerID="b5c29122bf9a1e7e05f51c5edbe97c2226837d683ce88addfb59fec43a6a4be0" Jul 21 13:02:34.146114 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:02:34.146093 2574 scope.go:117] "RemoveContainer" containerID="e61b058398072d494d3ebadc77335a04eff7e0bf549ed1169bbb55f0c15c7c89" Jul 21 13:03:13.335979 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.335924 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4"] Jul 21 13:03:13.336471 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.336330 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1283f3ad-e81e-43fe-ac59-1d3263fa58ea" containerName="maas-api" Jul 21 13:03:13.336471 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.336343 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1283f3ad-e81e-43fe-ac59-1d3263fa58ea" containerName="maas-api" Jul 21 13:03:13.336471 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.336358 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b7614a1-2bc9-458d-b596-2282d3b6d146" containerName="revoke-keys" Jul 21 13:03:13.336471 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.336364 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b7614a1-2bc9-458d-b596-2282d3b6d146" containerName="revoke-keys" Jul 21 13:03:13.336471 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.336430 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b7614a1-2bc9-458d-b596-2282d3b6d146" containerName="revoke-keys" Jul 21 13:03:13.336471 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.336439 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1283f3ad-e81e-43fe-ac59-1d3263fa58ea" containerName="maas-api" Jul 21 13:03:13.339470 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.339455 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:03:13.343597 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.343567 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-52a2ba-serving-cert\"" Jul 21 13:03:13.356686 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.356657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4\" (UID: \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:03:13.356805 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.356729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4mv2\" (UniqueName: \"kubernetes.io/projected/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-kube-api-access-z4mv2\") pod \"maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4\" (UID: \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:03:13.356805 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.356794 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-maas-api-tls\") pod \"maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4\" (UID: \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:03:13.367278 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.367251 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4"] Jul 21 13:03:13.458272 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.458230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4mv2\" (UniqueName: \"kubernetes.io/projected/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-kube-api-access-z4mv2\") pod \"maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4\" (UID: \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:03:13.458523 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.458317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-maas-api-tls\") pod \"maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4\" (UID: \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:03:13.458523 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.458394 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4\" (UID: \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:03:13.458523 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:03:13.458477 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-serving-cert: secret "maas-api-e2e-inf-a-52a2ba-serving-cert" not found Jul 21 13:03:13.458708 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:03:13.458567 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-maas-api-tls podName:c7b7a9f4-55b8-4262-88c2-1883f837a5d8 nodeName:}" failed. No retries permitted until 2026-07-21 13:03:13.958540091 +0000 UTC m=+2140.608953440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-maas-api-tls") pod "maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" (UID: "c7b7a9f4-55b8-4262-88c2-1883f837a5d8") : secret "maas-api-e2e-inf-a-52a2ba-serving-cert" not found Jul 21 13:03:13.459096 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.459071 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4\" (UID: \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:03:13.476112 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.476081 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4mv2\" (UniqueName: \"kubernetes.io/projected/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-kube-api-access-z4mv2\") pod \"maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4\" (UID: \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:03:13.962773 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.962739 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-maas-api-tls\") pod \"maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4\" (UID: \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:03:13.965345 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:13.965319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-maas-api-tls\") pod \"maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4\" (UID: \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:03:14.250004 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:14.249914 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:03:14.398035 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:14.398010 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4"] Jul 21 13:03:14.399721 ip-10-0-138-11 kubenswrapper[2574]: W0721 13:03:14.399697 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b7a9f4_55b8_4262_88c2_1883f837a5d8.slice/crio-eafd15344a57ae632e03dc3e132cf976cd97f0f927f955a9550a70dbf396ca23 WatchSource:0}: Error finding container eafd15344a57ae632e03dc3e132cf976cd97f0f927f955a9550a70dbf396ca23: Status 404 returned error can't find the container with id eafd15344a57ae632e03dc3e132cf976cd97f0f927f955a9550a70dbf396ca23 Jul 21 13:03:15.166085 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:15.165997 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" event={"ID":"c7b7a9f4-55b8-4262-88c2-1883f837a5d8","Type":"ContainerStarted","Data":"63905cc65b507f01d86be779833da329dc5026610016e78bc424301f1ada3410"} Jul 21 13:03:15.166085 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:15.166050 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" event={"ID":"c7b7a9f4-55b8-4262-88c2-1883f837a5d8","Type":"ContainerStarted","Data":"eafd15344a57ae632e03dc3e132cf976cd97f0f927f955a9550a70dbf396ca23"} Jul 21 13:03:15.166284 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:15.166143 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:03:15.186252 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:15.186204 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" podStartSLOduration=1.6888064109999998 podStartE2EDuration="2.186187959s" totalImagesPullingTime="497.381548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:03:15.183138395 +0000 UTC m=+2141.833551758" watchObservedRunningTime="2026-07-21 13:03:15.186187959 +0000 UTC m=+2141.836601320" Jul 21 13:03:21.176763 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:21.176734 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:03:21.455881 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:21.455806 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv"] Jul 21 13:03:21.459677 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:21.459656 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:03:21.463295 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:21.463273 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0bdb53-serving-cert\"" Jul 21 13:03:21.507292 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:21.507264 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv"] Jul 21 13:03:21.533326 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:21.533294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-maas-api-tls\") pod \"maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv\" (UID: \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:03:21.533481 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:21.533350 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnbdh\" (UniqueName: \"kubernetes.io/projected/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-kube-api-access-hnbdh\") pod \"maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv\" (UID: \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:03:21.533481 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:21.533458 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv\" (UID: \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:03:21.634019 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:21.633986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-maas-api-tls\") pod \"maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv\" (UID: \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:03:21.634019 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:21.634019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnbdh\" (UniqueName: \"kubernetes.io/projected/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-kube-api-access-hnbdh\") pod \"maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv\" (UID: \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:03:21.634230 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:21.634108 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv\" (UID: \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:03:21.634336 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:03:21.634316 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-serving-cert: secret "maas-api-e2e-inf-b-0bdb53-serving-cert" not found Jul 21 13:03:21.634416 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:03:21.634406 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-maas-api-tls podName:35983fac-9d6d-4602-8b8a-78cad5cfb0e1 nodeName:}" failed. No retries permitted until 2026-07-21 13:03:22.134381392 +0000 UTC m=+2148.784794751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-maas-api-tls") pod "maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" (UID: "35983fac-9d6d-4602-8b8a-78cad5cfb0e1") : secret "maas-api-e2e-inf-b-0bdb53-serving-cert" not found Jul 21 13:03:21.634746 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:21.634728 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv\" (UID: \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:03:21.650241 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:21.650213 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnbdh\" (UniqueName: \"kubernetes.io/projected/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-kube-api-access-hnbdh\") pod \"maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv\" (UID: \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:03:22.138399 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:22.138364 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-maas-api-tls\") pod \"maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv\" (UID: \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:03:22.140722 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:22.140692 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-maas-api-tls\") pod \"maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv\" (UID: \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:03:22.369906 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:22.369871 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:03:22.519469 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:22.519444 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv"] Jul 21 13:03:22.521311 ip-10-0-138-11 kubenswrapper[2574]: W0721 13:03:22.521284 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35983fac_9d6d_4602_8b8a_78cad5cfb0e1.slice/crio-9a93099e4342288877a4acf98f2e265ba65f59298ffc0bed5d2bc341b2420592 WatchSource:0}: Error finding container 9a93099e4342288877a4acf98f2e265ba65f59298ffc0bed5d2bc341b2420592: Status 404 returned error can't find the container with id 9a93099e4342288877a4acf98f2e265ba65f59298ffc0bed5d2bc341b2420592 Jul 21 13:03:23.210208 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:23.210176 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" event={"ID":"35983fac-9d6d-4602-8b8a-78cad5cfb0e1","Type":"ContainerStarted","Data":"794e419da3576371745908ce585a1aa08ec79b2c09276aaeedbaf6901b5a7fff"} Jul 21 13:03:23.210425 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:23.210224 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" event={"ID":"35983fac-9d6d-4602-8b8a-78cad5cfb0e1","Type":"ContainerStarted","Data":"9a93099e4342288877a4acf98f2e265ba65f59298ffc0bed5d2bc341b2420592"} Jul 21 13:03:23.210425 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:23.210257 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:03:23.227594 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:23.227543 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" podStartSLOduration=1.6636976959999998 podStartE2EDuration="2.227527772s" totalImagesPullingTime="563.830076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:03:23.226548916 +0000 UTC m=+2149.876962287" watchObservedRunningTime="2026-07-21 13:03:23.227527772 +0000 UTC m=+2149.877941132" Jul 21 13:03:29.221486 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:29.221413 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:03:34.171193 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:03:34.171162 2574 scope.go:117] "RemoveContainer" containerID="d58daabde643092a30fbc8ba5737f818dd4230cae5f3a2e397cf4ebdca2b027f" Jul 21 13:04:57.073188 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.073114 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4"] Jul 21 13:04:57.075535 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.073328 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" podUID="c7b7a9f4-55b8-4262-88c2-1883f837a5d8" containerName="maas-api" containerID="cri-o://63905cc65b507f01d86be779833da329dc5026610016e78bc424301f1ada3410" gracePeriod=30 Jul 21 13:04:57.324586 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.324525 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:04:57.444346 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.444311 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-openshift-service-ca\") pod \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\" (UID: \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\") " Jul 21 13:04:57.444537 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.444367 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4mv2\" (UniqueName: \"kubernetes.io/projected/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-kube-api-access-z4mv2\") pod \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\" (UID: \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\") " Jul 21 13:04:57.444537 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.444416 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-maas-api-tls\") pod \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\" (UID: \"c7b7a9f4-55b8-4262-88c2-1883f837a5d8\") " Jul 21 13:04:57.444728 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.444700 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7b7a9f4-55b8-4262-88c2-1883f837a5d8" (UID: "c7b7a9f4-55b8-4262-88c2-1883f837a5d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:04:57.446583 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.446560 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-kube-api-access-z4mv2" (OuterVolumeSpecName: "kube-api-access-z4mv2") pod "c7b7a9f4-55b8-4262-88c2-1883f837a5d8" (UID: "c7b7a9f4-55b8-4262-88c2-1883f837a5d8"). InnerVolumeSpecName "kube-api-access-z4mv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:04:57.446583 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.446572 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7b7a9f4-55b8-4262-88c2-1883f837a5d8" (UID: "c7b7a9f4-55b8-4262-88c2-1883f837a5d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:04:57.545902 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.545870 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-openshift-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:04:57.545902 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.545896 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4mv2\" (UniqueName: \"kubernetes.io/projected/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-kube-api-access-z4mv2\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:04:57.545902 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.545906 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b7a9f4-55b8-4262-88c2-1883f837a5d8-maas-api-tls\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:04:57.719714 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.719624 2574 generic.go:356] "Generic (PLEG): container finished" podID="c7b7a9f4-55b8-4262-88c2-1883f837a5d8" containerID="63905cc65b507f01d86be779833da329dc5026610016e78bc424301f1ada3410" exitCode=0 Jul 21 13:04:57.719714 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.719676 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" Jul 21 13:04:57.719896 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.719712 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" event={"ID":"c7b7a9f4-55b8-4262-88c2-1883f837a5d8","Type":"ContainerDied","Data":"63905cc65b507f01d86be779833da329dc5026610016e78bc424301f1ada3410"} Jul 21 13:04:57.719896 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.719761 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4" event={"ID":"c7b7a9f4-55b8-4262-88c2-1883f837a5d8","Type":"ContainerDied","Data":"eafd15344a57ae632e03dc3e132cf976cd97f0f927f955a9550a70dbf396ca23"} Jul 21 13:04:57.719896 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.719786 2574 scope.go:117] "RemoveContainer" containerID="63905cc65b507f01d86be779833da329dc5026610016e78bc424301f1ada3410" Jul 21 13:04:57.729694 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.729677 2574 scope.go:117] "RemoveContainer" containerID="63905cc65b507f01d86be779833da329dc5026610016e78bc424301f1ada3410" Jul 21 13:04:57.729932 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:04:57.729906 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63905cc65b507f01d86be779833da329dc5026610016e78bc424301f1ada3410\": container with ID starting with 63905cc65b507f01d86be779833da329dc5026610016e78bc424301f1ada3410 not found: ID does not exist" containerID="63905cc65b507f01d86be779833da329dc5026610016e78bc424301f1ada3410" Jul 21 13:04:57.730061 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.729952 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63905cc65b507f01d86be779833da329dc5026610016e78bc424301f1ada3410"} err="failed to get container status \"63905cc65b507f01d86be779833da329dc5026610016e78bc424301f1ada3410\": rpc error: code = NotFound desc = could not find container \"63905cc65b507f01d86be779833da329dc5026610016e78bc424301f1ada3410\": container with ID starting with 63905cc65b507f01d86be779833da329dc5026610016e78bc424301f1ada3410 not found: ID does not exist" Jul 21 13:04:57.744437 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.744416 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4"] Jul 21 13:04:57.749114 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.749094 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52a2ba-96c986cbd-m76c4"] Jul 21 13:04:57.925997 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:04:57.925961 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7b7a9f4-55b8-4262-88c2-1883f837a5d8" path="/var/lib/kubelet/pods/c7b7a9f4-55b8-4262-88c2-1883f837a5d8/volumes" Jul 21 13:05:08.505159 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:08.505121 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2"] Jul 21 13:05:08.505637 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:08.505510 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b7a9f4-55b8-4262-88c2-1883f837a5d8" containerName="maas-api" Jul 21 13:05:08.505637 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:08.505521 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b7a9f4-55b8-4262-88c2-1883f837a5d8" containerName="maas-api" Jul 21 13:05:08.505637 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:08.505578 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b7a9f4-55b8-4262-88c2-1883f837a5d8" containerName="maas-api" Jul 21 13:05:08.508627 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:08.508610 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2" Jul 21 13:05:08.511459 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:08.511437 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jq8px\"" Jul 21 13:05:08.515523 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:08.515498 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2"] Jul 21 13:05:08.543809 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:08.543778 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1414a2-c1fe-4416-8e09-7672dc7bf725-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2\" (UID: \"0f1414a2-c1fe-4416-8e09-7672dc7bf725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2" Jul 21 13:05:08.644368 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:08.644328 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1414a2-c1fe-4416-8e09-7672dc7bf725-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2\" (UID: \"0f1414a2-c1fe-4416-8e09-7672dc7bf725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2" Jul 21 13:05:08.644928 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:08.644907 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1414a2-c1fe-4416-8e09-7672dc7bf725-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2\" (UID: \"0f1414a2-c1fe-4416-8e09-7672dc7bf725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2" Jul 21 13:05:08.821713 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:08.821616 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2" Jul 21 13:05:08.969158 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:08.969132 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2"] Jul 21 13:05:08.970932 ip-10-0-138-11 kubenswrapper[2574]: W0721 13:05:08.970904 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1414a2_c1fe_4416_8e09_7672dc7bf725.slice/crio-e1e3eea36767d2b00a7c2174b12f014890bad6cf5b22b03bb203e3874d3e600d WatchSource:0}: Error finding container e1e3eea36767d2b00a7c2174b12f014890bad6cf5b22b03bb203e3874d3e600d: Status 404 returned error can't find the container with id e1e3eea36767d2b00a7c2174b12f014890bad6cf5b22b03bb203e3874d3e600d Jul 21 13:05:09.791655 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:09.791624 2574 generic.go:356] "Generic (PLEG): container finished" podID="0f1414a2-c1fe-4416-8e09-7672dc7bf725" containerID="ce152725dede81c2f39a9479339ac4fcea1d4cb5d39f995dc300e95a122bb62c" exitCode=0 Jul 21 13:05:09.792052 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:09.791712 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2" event={"ID":"0f1414a2-c1fe-4416-8e09-7672dc7bf725","Type":"ContainerDied","Data":"ce152725dede81c2f39a9479339ac4fcea1d4cb5d39f995dc300e95a122bb62c"} Jul 21 13:05:09.792052 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:09.791769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2" event={"ID":"0f1414a2-c1fe-4416-8e09-7672dc7bf725","Type":"ContainerStarted","Data":"e1e3eea36767d2b00a7c2174b12f014890bad6cf5b22b03bb203e3874d3e600d"} Jul 21 13:05:10.939888 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:10.939860 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2" Jul 21 13:05:10.969529 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:10.964267 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1414a2-c1fe-4416-8e09-7672dc7bf725-maas-api-service-ca\") pod \"0f1414a2-c1fe-4416-8e09-7672dc7bf725\" (UID: \"0f1414a2-c1fe-4416-8e09-7672dc7bf725\") " Jul 21 13:05:10.969529 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:10.969194 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1414a2-c1fe-4416-8e09-7672dc7bf725-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f1414a2-c1fe-4416-8e09-7672dc7bf725" (UID: "0f1414a2-c1fe-4416-8e09-7672dc7bf725"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:05:11.065544 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:11.065452 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1414a2-c1fe-4416-8e09-7672dc7bf725-maas-api-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:05:11.803151 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:11.803115 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2" event={"ID":"0f1414a2-c1fe-4416-8e09-7672dc7bf725","Type":"ContainerDied","Data":"e1e3eea36767d2b00a7c2174b12f014890bad6cf5b22b03bb203e3874d3e600d"} Jul 21 13:05:11.803151 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:11.803146 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2" Jul 21 13:05:11.803363 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:11.803190 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1e3eea36767d2b00a7c2174b12f014890bad6cf5b22b03bb203e3874d3e600d" Jul 21 13:05:18.996406 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:18.996369 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv"] Jul 21 13:05:18.996797 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:18.996601 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" podUID="35983fac-9d6d-4602-8b8a-78cad5cfb0e1" containerName="maas-api" containerID="cri-o://794e419da3576371745908ce585a1aa08ec79b2c09276aaeedbaf6901b5a7fff" gracePeriod=30 Jul 21 13:05:19.259371 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.259344 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:05:19.338096 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.338064 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnbdh\" (UniqueName: \"kubernetes.io/projected/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-kube-api-access-hnbdh\") pod \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\" (UID: \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\") " Jul 21 13:05:19.338096 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.338098 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-openshift-service-ca\") pod \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\" (UID: \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\") " Jul 21 13:05:19.338357 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.338126 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-maas-api-tls\") pod \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\" (UID: \"35983fac-9d6d-4602-8b8a-78cad5cfb0e1\") " Jul 21 13:05:19.338567 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.338538 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "35983fac-9d6d-4602-8b8a-78cad5cfb0e1" (UID: "35983fac-9d6d-4602-8b8a-78cad5cfb0e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:05:19.340310 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.340281 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "35983fac-9d6d-4602-8b8a-78cad5cfb0e1" (UID: "35983fac-9d6d-4602-8b8a-78cad5cfb0e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:05:19.340402 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.340348 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-kube-api-access-hnbdh" (OuterVolumeSpecName: "kube-api-access-hnbdh") pod "35983fac-9d6d-4602-8b8a-78cad5cfb0e1" (UID: "35983fac-9d6d-4602-8b8a-78cad5cfb0e1"). InnerVolumeSpecName "kube-api-access-hnbdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:05:19.438727 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.438695 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-maas-api-tls\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:05:19.438727 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.438726 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnbdh\" (UniqueName: \"kubernetes.io/projected/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-kube-api-access-hnbdh\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:05:19.438953 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.438740 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35983fac-9d6d-4602-8b8a-78cad5cfb0e1-openshift-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:05:19.850370 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.850332 2574 generic.go:356] "Generic (PLEG): container finished" podID="35983fac-9d6d-4602-8b8a-78cad5cfb0e1" containerID="794e419da3576371745908ce585a1aa08ec79b2c09276aaeedbaf6901b5a7fff" exitCode=0 Jul 21 13:05:19.850560 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.850373 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" Jul 21 13:05:19.850560 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.850388 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" event={"ID":"35983fac-9d6d-4602-8b8a-78cad5cfb0e1","Type":"ContainerDied","Data":"794e419da3576371745908ce585a1aa08ec79b2c09276aaeedbaf6901b5a7fff"} Jul 21 13:05:19.850560 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.850436 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" event={"ID":"35983fac-9d6d-4602-8b8a-78cad5cfb0e1","Type":"ContainerDied","Data":"9a93099e4342288877a4acf98f2e265ba65f59298ffc0bed5d2bc341b2420592"} Jul 21 13:05:19.850560 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.850459 2574 scope.go:117] "RemoveContainer" containerID="794e419da3576371745908ce585a1aa08ec79b2c09276aaeedbaf6901b5a7fff" Jul 21 13:05:19.860549 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.860529 2574 scope.go:117] "RemoveContainer" containerID="794e419da3576371745908ce585a1aa08ec79b2c09276aaeedbaf6901b5a7fff" Jul 21 13:05:19.860826 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:05:19.860805 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794e419da3576371745908ce585a1aa08ec79b2c09276aaeedbaf6901b5a7fff\": container with ID starting with 794e419da3576371745908ce585a1aa08ec79b2c09276aaeedbaf6901b5a7fff not found: ID does not exist" containerID="794e419da3576371745908ce585a1aa08ec79b2c09276aaeedbaf6901b5a7fff" Jul 21 13:05:19.860867 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.860836 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794e419da3576371745908ce585a1aa08ec79b2c09276aaeedbaf6901b5a7fff"} err="failed to get container status \"794e419da3576371745908ce585a1aa08ec79b2c09276aaeedbaf6901b5a7fff\": rpc error: code = NotFound desc = could not find container \"794e419da3576371745908ce585a1aa08ec79b2c09276aaeedbaf6901b5a7fff\": container with ID starting with 794e419da3576371745908ce585a1aa08ec79b2c09276aaeedbaf6901b5a7fff not found: ID does not exist" Jul 21 13:05:19.875201 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.875168 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv"] Jul 21 13:05:19.881321 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.881293 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv"] Jul 21 13:05:19.924491 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:19.924456 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35983fac-9d6d-4602-8b8a-78cad5cfb0e1" path="/var/lib/kubelet/pods/35983fac-9d6d-4602-8b8a-78cad5cfb0e1/volumes" Jul 21 13:05:22.216498 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:22.216435 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0bdb53-7555db678-w6rmv" podUID="35983fac-9d6d-4602-8b8a-78cad5cfb0e1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.60:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 21 13:05:23.571121 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:23.571079 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2"] Jul 21 13:05:23.574142 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:23.574117 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bdb53-mt5z2"] Jul 21 13:05:23.925549 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:23.925471 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f1414a2-c1fe-4416-8e09-7672dc7bf725" path="/var/lib/kubelet/pods/0f1414a2-c1fe-4416-8e09-7672dc7bf725/volumes" Jul 21 13:05:34.017571 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.017535 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9"] Jul 21 13:05:34.018300 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.017930 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35983fac-9d6d-4602-8b8a-78cad5cfb0e1" containerName="maas-api" Jul 21 13:05:34.018300 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.017955 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="35983fac-9d6d-4602-8b8a-78cad5cfb0e1" containerName="maas-api" Jul 21 13:05:34.018300 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.017963 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1414a2-c1fe-4416-8e09-7672dc7bf725" containerName="revoke-keys" Jul 21 13:05:34.018300 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.017968 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1414a2-c1fe-4416-8e09-7672dc7bf725" containerName="revoke-keys" Jul 21 13:05:34.018300 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.018022 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="35983fac-9d6d-4602-8b8a-78cad5cfb0e1" containerName="maas-api" Jul 21 13:05:34.018300 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.018031 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1414a2-c1fe-4416-8e09-7672dc7bf725" containerName="revoke-keys" Jul 21 13:05:34.022323 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.022300 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:05:34.024789 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.024767 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-019b5c-serving-cert\"" Jul 21 13:05:34.031486 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.031459 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9"] Jul 21 13:05:34.073215 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.073177 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/677def94-1f6e-495c-8e5e-454ec9619758-maas-api-tls\") pod \"maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9\" (UID: \"677def94-1f6e-495c-8e5e-454ec9619758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:05:34.073389 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.073223 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjfzl\" (UniqueName: \"kubernetes.io/projected/677def94-1f6e-495c-8e5e-454ec9619758-kube-api-access-xjfzl\") pod \"maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9\" (UID: \"677def94-1f6e-495c-8e5e-454ec9619758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:05:34.073389 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.073272 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/677def94-1f6e-495c-8e5e-454ec9619758-openshift-service-ca\") pod \"maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9\" (UID: \"677def94-1f6e-495c-8e5e-454ec9619758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:05:34.174178 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.174144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/677def94-1f6e-495c-8e5e-454ec9619758-maas-api-tls\") pod \"maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9\" (UID: \"677def94-1f6e-495c-8e5e-454ec9619758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:05:34.174178 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.174189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjfzl\" (UniqueName: \"kubernetes.io/projected/677def94-1f6e-495c-8e5e-454ec9619758-kube-api-access-xjfzl\") pod \"maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9\" (UID: \"677def94-1f6e-495c-8e5e-454ec9619758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:05:34.174412 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:05:34.174284 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-serving-cert: secret "maas-api-e2e-api-a-019b5c-serving-cert" not found Jul 21 13:05:34.174412 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.174309 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/677def94-1f6e-495c-8e5e-454ec9619758-openshift-service-ca\") pod \"maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9\" (UID: \"677def94-1f6e-495c-8e5e-454ec9619758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:05:34.174412 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:05:34.174358 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677def94-1f6e-495c-8e5e-454ec9619758-maas-api-tls podName:677def94-1f6e-495c-8e5e-454ec9619758 nodeName:}" failed. No retries permitted until 2026-07-21 13:05:34.674339768 +0000 UTC m=+2281.324753107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/677def94-1f6e-495c-8e5e-454ec9619758-maas-api-tls") pod "maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" (UID: "677def94-1f6e-495c-8e5e-454ec9619758") : secret "maas-api-e2e-api-a-019b5c-serving-cert" not found Jul 21 13:05:34.174909 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.174891 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/677def94-1f6e-495c-8e5e-454ec9619758-openshift-service-ca\") pod \"maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9\" (UID: \"677def94-1f6e-495c-8e5e-454ec9619758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:05:34.182807 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.182776 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjfzl\" (UniqueName: \"kubernetes.io/projected/677def94-1f6e-495c-8e5e-454ec9619758-kube-api-access-xjfzl\") pod \"maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9\" (UID: \"677def94-1f6e-495c-8e5e-454ec9619758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:05:34.200027 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.199998 2574 scope.go:117] "RemoveContainer" containerID="234ad54fb07a1b018639956b98190fd460d6215d0a95b97cf186e2b97b25b902" Jul 21 13:05:34.679460 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.679417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/677def94-1f6e-495c-8e5e-454ec9619758-maas-api-tls\") pod \"maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9\" (UID: \"677def94-1f6e-495c-8e5e-454ec9619758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:05:34.681897 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.681869 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/677def94-1f6e-495c-8e5e-454ec9619758-maas-api-tls\") pod \"maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9\" (UID: \"677def94-1f6e-495c-8e5e-454ec9619758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:05:34.934384 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:34.934300 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:05:35.083001 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:35.082965 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9"] Jul 21 13:05:35.084302 ip-10-0-138-11 kubenswrapper[2574]: W0721 13:05:35.084275 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677def94_1f6e_495c_8e5e_454ec9619758.slice/crio-a9855677086526459e1098c98dc66b14e770266373dcad689164b40afa7bdfb7 WatchSource:0}: Error finding container a9855677086526459e1098c98dc66b14e770266373dcad689164b40afa7bdfb7: Status 404 returned error can't find the container with id a9855677086526459e1098c98dc66b14e770266373dcad689164b40afa7bdfb7 Jul 21 13:05:35.085713 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:35.085694 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 13:05:35.942370 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:35.942334 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" event={"ID":"677def94-1f6e-495c-8e5e-454ec9619758","Type":"ContainerStarted","Data":"8d099235c9f74564c7a2f7b7b2e4eb36b726c6afbab63a8d51c06bfa03961c27"} Jul 21 13:05:35.942577 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:35.942398 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" event={"ID":"677def94-1f6e-495c-8e5e-454ec9619758","Type":"ContainerStarted","Data":"a9855677086526459e1098c98dc66b14e770266373dcad689164b40afa7bdfb7"} Jul 21 13:05:35.942577 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:35.942519 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:05:35.961988 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:35.961919 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" podStartSLOduration=2.284454096 podStartE2EDuration="2.961898571s" totalImagesPullingTime="677.444475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:05:35.959924709 +0000 UTC m=+2282.610338070" watchObservedRunningTime="2026-07-21 13:05:35.961898571 +0000 UTC m=+2282.612311935" Jul 21 13:05:41.953552 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:41.953521 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:05:42.090758 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:42.090727 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl"] Jul 21 13:05:42.094899 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:42.094873 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:05:42.097495 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:42.097472 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-69d4e8-serving-cert\"" Jul 21 13:05:42.105436 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:42.105405 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl"] Jul 21 13:05:42.253568 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:42.253525 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8bth\" (UniqueName: \"kubernetes.io/projected/d0c626e0-f3c5-453d-9582-434b8250d678-kube-api-access-v8bth\") pod \"maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl\" (UID: \"d0c626e0-f3c5-453d-9582-434b8250d678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:05:42.253767 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:42.253606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0c626e0-f3c5-453d-9582-434b8250d678-maas-api-tls\") pod \"maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl\" (UID: \"d0c626e0-f3c5-453d-9582-434b8250d678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:05:42.253767 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:42.253650 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c626e0-f3c5-453d-9582-434b8250d678-openshift-service-ca\") pod \"maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl\" (UID: \"d0c626e0-f3c5-453d-9582-434b8250d678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:05:42.354800 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:42.354758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0c626e0-f3c5-453d-9582-434b8250d678-maas-api-tls\") pod \"maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl\" (UID: \"d0c626e0-f3c5-453d-9582-434b8250d678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:05:42.354800 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:42.354801 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c626e0-f3c5-453d-9582-434b8250d678-openshift-service-ca\") pod \"maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl\" (UID: \"d0c626e0-f3c5-453d-9582-434b8250d678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:05:42.355077 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:42.354872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8bth\" (UniqueName: \"kubernetes.io/projected/d0c626e0-f3c5-453d-9582-434b8250d678-kube-api-access-v8bth\") pod \"maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl\" (UID: \"d0c626e0-f3c5-453d-9582-434b8250d678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:05:42.355077 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:05:42.354908 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-serving-cert: secret "maas-api-e2e-api-b-69d4e8-serving-cert" not found Jul 21 13:05:42.355077 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:05:42.355003 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0c626e0-f3c5-453d-9582-434b8250d678-maas-api-tls podName:d0c626e0-f3c5-453d-9582-434b8250d678 nodeName:}" failed. No retries permitted until 2026-07-21 13:05:42.854982456 +0000 UTC m=+2289.505395811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0c626e0-f3c5-453d-9582-434b8250d678-maas-api-tls") pod "maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" (UID: "d0c626e0-f3c5-453d-9582-434b8250d678") : secret "maas-api-e2e-api-b-69d4e8-serving-cert" not found Jul 21 13:05:42.355528 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:42.355507 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c626e0-f3c5-453d-9582-434b8250d678-openshift-service-ca\") pod \"maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl\" (UID: \"d0c626e0-f3c5-453d-9582-434b8250d678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:05:42.363790 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:42.363760 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8bth\" (UniqueName: \"kubernetes.io/projected/d0c626e0-f3c5-453d-9582-434b8250d678-kube-api-access-v8bth\") pod \"maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl\" (UID: \"d0c626e0-f3c5-453d-9582-434b8250d678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:05:42.860098 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:42.860052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0c626e0-f3c5-453d-9582-434b8250d678-maas-api-tls\") pod \"maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl\" (UID: \"d0c626e0-f3c5-453d-9582-434b8250d678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:05:42.862557 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:42.862524 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0c626e0-f3c5-453d-9582-434b8250d678-maas-api-tls\") pod \"maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl\" (UID: \"d0c626e0-f3c5-453d-9582-434b8250d678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:05:43.008205 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:43.008155 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:05:43.152422 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:43.152384 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl"] Jul 21 13:05:43.153813 ip-10-0-138-11 kubenswrapper[2574]: W0721 13:05:43.153781 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c626e0_f3c5_453d_9582_434b8250d678.slice/crio-20ad5cc832ce6ad2339af65c19a34d04be32709c10bcc706b19e39e10545eeb1 WatchSource:0}: Error finding container 20ad5cc832ce6ad2339af65c19a34d04be32709c10bcc706b19e39e10545eeb1: Status 404 returned error can't find the container with id 20ad5cc832ce6ad2339af65c19a34d04be32709c10bcc706b19e39e10545eeb1 Jul 21 13:05:43.990598 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:43.990492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" event={"ID":"d0c626e0-f3c5-453d-9582-434b8250d678","Type":"ContainerStarted","Data":"d69f48ecedce5d149bf9ef25d09d019df399fb8e4f7462f467dd658971504c16"} Jul 21 13:05:43.990598 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:43.990554 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" event={"ID":"d0c626e0-f3c5-453d-9582-434b8250d678","Type":"ContainerStarted","Data":"20ad5cc832ce6ad2339af65c19a34d04be32709c10bcc706b19e39e10545eeb1"} Jul 21 13:05:43.990825 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:43.990641 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:05:44.008624 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:44.008567 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" podStartSLOduration=1.641123669 podStartE2EDuration="2.008548422s" totalImagesPullingTime="367.424753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:05:44.007192576 +0000 UTC m=+2290.657605937" watchObservedRunningTime="2026-07-21 13:05:44.008548422 +0000 UTC m=+2290.658961783" Jul 21 13:05:50.003416 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:50.003388 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:05:54.794207 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:54.794165 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv"] Jul 21 13:05:54.797907 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:54.797889 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv" Jul 21 13:05:54.800512 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:54.800490 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jq8px\"" Jul 21 13:05:54.807010 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:54.806201 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv"] Jul 21 13:05:54.872180 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:54.872147 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2d7769-572c-4edf-9f65-e2b7903eb7fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv\" (UID: \"3b2d7769-572c-4edf-9f65-e2b7903eb7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv" Jul 21 13:05:54.972830 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:54.972799 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2d7769-572c-4edf-9f65-e2b7903eb7fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv\" (UID: \"3b2d7769-572c-4edf-9f65-e2b7903eb7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv" Jul 21 13:05:54.973469 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:54.973447 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2d7769-572c-4edf-9f65-e2b7903eb7fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv\" (UID: \"3b2d7769-572c-4edf-9f65-e2b7903eb7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv" Jul 21 13:05:55.110480 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:55.110395 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv" Jul 21 13:05:55.248310 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:55.248282 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv"] Jul 21 13:05:55.250337 ip-10-0-138-11 kubenswrapper[2574]: W0721 13:05:55.250308 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2d7769_572c_4edf_9f65_e2b7903eb7fd.slice/crio-ef948b43796b452392ce34bb44cb81ad157000686ab777a21b0b2c43c612a139 WatchSource:0}: Error finding container ef948b43796b452392ce34bb44cb81ad157000686ab777a21b0b2c43c612a139: Status 404 returned error can't find the container with id ef948b43796b452392ce34bb44cb81ad157000686ab777a21b0b2c43c612a139 Jul 21 13:05:56.059083 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:56.059055 2574 generic.go:356] "Generic (PLEG): container finished" podID="3b2d7769-572c-4edf-9f65-e2b7903eb7fd" containerID="7344aa46c0ade8e23026d9b44e5378f6e42269c160eac56eefbc2730702e7124" exitCode=0 Jul 21 13:05:56.059416 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:56.059132 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv" event={"ID":"3b2d7769-572c-4edf-9f65-e2b7903eb7fd","Type":"ContainerDied","Data":"7344aa46c0ade8e23026d9b44e5378f6e42269c160eac56eefbc2730702e7124"} Jul 21 13:05:56.059416 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:56.059183 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv" event={"ID":"3b2d7769-572c-4edf-9f65-e2b7903eb7fd","Type":"ContainerStarted","Data":"ef948b43796b452392ce34bb44cb81ad157000686ab777a21b0b2c43c612a139"} Jul 21 13:05:57.203969 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:57.203929 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv" Jul 21 13:05:57.288597 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:57.288554 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2d7769-572c-4edf-9f65-e2b7903eb7fd-maas-api-service-ca\") pod \"3b2d7769-572c-4edf-9f65-e2b7903eb7fd\" (UID: \"3b2d7769-572c-4edf-9f65-e2b7903eb7fd\") " Jul 21 13:05:57.289024 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:57.289003 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2d7769-572c-4edf-9f65-e2b7903eb7fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b2d7769-572c-4edf-9f65-e2b7903eb7fd" (UID: "3b2d7769-572c-4edf-9f65-e2b7903eb7fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:05:57.389914 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:57.389823 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2d7769-572c-4edf-9f65-e2b7903eb7fd-maas-api-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:05:58.071537 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:58.071508 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv" Jul 21 13:05:58.071712 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:58.071560 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv" event={"ID":"3b2d7769-572c-4edf-9f65-e2b7903eb7fd","Type":"ContainerDied","Data":"ef948b43796b452392ce34bb44cb81ad157000686ab777a21b0b2c43c612a139"} Jul 21 13:05:58.071712 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:05:58.071613 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef948b43796b452392ce34bb44cb81ad157000686ab777a21b0b2c43c612a139" Jul 21 13:06:10.671227 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:10.671190 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9"] Jul 21 13:06:10.671843 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:10.671635 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" podUID="677def94-1f6e-495c-8e5e-454ec9619758" containerName="maas-api" containerID="cri-o://8d099235c9f74564c7a2f7b7b2e4eb36b726c6afbab63a8d51c06bfa03961c27" gracePeriod=30 Jul 21 13:06:10.930458 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:10.930395 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:06:11.006697 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.006664 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/677def94-1f6e-495c-8e5e-454ec9619758-maas-api-tls\") pod \"677def94-1f6e-495c-8e5e-454ec9619758\" (UID: \"677def94-1f6e-495c-8e5e-454ec9619758\") " Jul 21 13:06:11.006862 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.006763 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjfzl\" (UniqueName: \"kubernetes.io/projected/677def94-1f6e-495c-8e5e-454ec9619758-kube-api-access-xjfzl\") pod \"677def94-1f6e-495c-8e5e-454ec9619758\" (UID: \"677def94-1f6e-495c-8e5e-454ec9619758\") " Jul 21 13:06:11.006862 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.006838 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/677def94-1f6e-495c-8e5e-454ec9619758-openshift-service-ca\") pod \"677def94-1f6e-495c-8e5e-454ec9619758\" (UID: \"677def94-1f6e-495c-8e5e-454ec9619758\") " Jul 21 13:06:11.007225 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.007196 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677def94-1f6e-495c-8e5e-454ec9619758-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "677def94-1f6e-495c-8e5e-454ec9619758" (UID: "677def94-1f6e-495c-8e5e-454ec9619758"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:06:11.008794 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.008768 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677def94-1f6e-495c-8e5e-454ec9619758-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "677def94-1f6e-495c-8e5e-454ec9619758" (UID: "677def94-1f6e-495c-8e5e-454ec9619758"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:06:11.008881 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.008865 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677def94-1f6e-495c-8e5e-454ec9619758-kube-api-access-xjfzl" (OuterVolumeSpecName: "kube-api-access-xjfzl") pod "677def94-1f6e-495c-8e5e-454ec9619758" (UID: "677def94-1f6e-495c-8e5e-454ec9619758"). InnerVolumeSpecName "kube-api-access-xjfzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:06:11.108589 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.108552 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/677def94-1f6e-495c-8e5e-454ec9619758-openshift-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:06:11.108589 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.108582 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/677def94-1f6e-495c-8e5e-454ec9619758-maas-api-tls\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:06:11.108589 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.108593 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjfzl\" (UniqueName: \"kubernetes.io/projected/677def94-1f6e-495c-8e5e-454ec9619758-kube-api-access-xjfzl\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:06:11.144179 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.144145 2574 generic.go:356] "Generic (PLEG): container finished" podID="677def94-1f6e-495c-8e5e-454ec9619758" containerID="8d099235c9f74564c7a2f7b7b2e4eb36b726c6afbab63a8d51c06bfa03961c27" exitCode=0 Jul 21 13:06:11.144339 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.144243 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" Jul 21 13:06:11.144339 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.144256 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" event={"ID":"677def94-1f6e-495c-8e5e-454ec9619758","Type":"ContainerDied","Data":"8d099235c9f74564c7a2f7b7b2e4eb36b726c6afbab63a8d51c06bfa03961c27"} Jul 21 13:06:11.144339 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.144298 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9" event={"ID":"677def94-1f6e-495c-8e5e-454ec9619758","Type":"ContainerDied","Data":"a9855677086526459e1098c98dc66b14e770266373dcad689164b40afa7bdfb7"} Jul 21 13:06:11.144339 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.144318 2574 scope.go:117] "RemoveContainer" containerID="8d099235c9f74564c7a2f7b7b2e4eb36b726c6afbab63a8d51c06bfa03961c27" Jul 21 13:06:11.155187 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.155168 2574 scope.go:117] "RemoveContainer" containerID="8d099235c9f74564c7a2f7b7b2e4eb36b726c6afbab63a8d51c06bfa03961c27" Jul 21 13:06:11.155475 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:06:11.155455 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d099235c9f74564c7a2f7b7b2e4eb36b726c6afbab63a8d51c06bfa03961c27\": container with ID starting with 8d099235c9f74564c7a2f7b7b2e4eb36b726c6afbab63a8d51c06bfa03961c27 not found: ID does not exist" containerID="8d099235c9f74564c7a2f7b7b2e4eb36b726c6afbab63a8d51c06bfa03961c27" Jul 21 13:06:11.155525 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.155487 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d099235c9f74564c7a2f7b7b2e4eb36b726c6afbab63a8d51c06bfa03961c27"} err="failed to get container status \"8d099235c9f74564c7a2f7b7b2e4eb36b726c6afbab63a8d51c06bfa03961c27\": rpc error: code = NotFound desc = could not find container \"8d099235c9f74564c7a2f7b7b2e4eb36b726c6afbab63a8d51c06bfa03961c27\": container with ID starting with 8d099235c9f74564c7a2f7b7b2e4eb36b726c6afbab63a8d51c06bfa03961c27 not found: ID does not exist" Jul 21 13:06:11.167843 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.167815 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9"] Jul 21 13:06:11.171774 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.171752 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-019b5c-7f79f8cfb4-c5tb9"] Jul 21 13:06:11.180116 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:06:11.180094 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677def94_1f6e_495c_8e5e_454ec9619758.slice\": RecentStats: unable to find data in memory cache]" Jul 21 13:06:11.924695 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:11.924664 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="677def94-1f6e-495c-8e5e-454ec9619758" path="/var/lib/kubelet/pods/677def94-1f6e-495c-8e5e-454ec9619758/volumes" Jul 21 13:06:14.882291 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:14.882253 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv"] Jul 21 13:06:14.884479 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:14.884455 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-019b5c-pd6bv"] Jul 21 13:06:15.924407 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:15.924376 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b2d7769-572c-4edf-9f65-e2b7903eb7fd" path="/var/lib/kubelet/pods/3b2d7769-572c-4edf-9f65-e2b7903eb7fd/volumes" Jul 21 13:06:21.840134 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:21.840095 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv"] Jul 21 13:06:21.840593 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:21.840478 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="677def94-1f6e-495c-8e5e-454ec9619758" containerName="maas-api" Jul 21 13:06:21.840593 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:21.840488 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="677def94-1f6e-495c-8e5e-454ec9619758" containerName="maas-api" Jul 21 13:06:21.840593 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:21.840498 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2d7769-572c-4edf-9f65-e2b7903eb7fd" containerName="revoke-keys" Jul 21 13:06:21.840593 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:21.840504 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2d7769-572c-4edf-9f65-e2b7903eb7fd" containerName="revoke-keys" Jul 21 13:06:21.840738 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:21.840599 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="677def94-1f6e-495c-8e5e-454ec9619758" containerName="maas-api" Jul 21 13:06:21.840738 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:21.840607 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b2d7769-572c-4edf-9f65-e2b7903eb7fd" containerName="revoke-keys" Jul 21 13:06:21.845023 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:21.845007 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv" Jul 21 13:06:21.847955 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:21.847914 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jq8px\"" Jul 21 13:06:21.851293 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:21.851266 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv"] Jul 21 13:06:21.904610 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:21.904579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f85524-b092-454a-8d41-baaa24b75709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv\" (UID: \"f9f85524-b092-454a-8d41-baaa24b75709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv" Jul 21 13:06:22.005519 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:22.005482 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f85524-b092-454a-8d41-baaa24b75709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv\" (UID: \"f9f85524-b092-454a-8d41-baaa24b75709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv" Jul 21 13:06:22.006303 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:22.006268 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f85524-b092-454a-8d41-baaa24b75709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv\" (UID: \"f9f85524-b092-454a-8d41-baaa24b75709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv" Jul 21 13:06:22.158914 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:22.158818 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv" Jul 21 13:06:22.299187 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:22.299131 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv"] Jul 21 13:06:22.301601 ip-10-0-138-11 kubenswrapper[2574]: W0721 13:06:22.301560 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f85524_b092_454a_8d41_baaa24b75709.slice/crio-d6740e05bb4c69741f1c7f70055e04beffde6695abf6263978d4b29b9a80645a WatchSource:0}: Error finding container d6740e05bb4c69741f1c7f70055e04beffde6695abf6263978d4b29b9a80645a: Status 404 returned error can't find the container with id d6740e05bb4c69741f1c7f70055e04beffde6695abf6263978d4b29b9a80645a Jul 21 13:06:23.212965 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:23.212851 2574 generic.go:356] "Generic (PLEG): container finished" podID="f9f85524-b092-454a-8d41-baaa24b75709" containerID="8ab3cce8498908650152f03ac248f950c0c4eb399234b5b183201c688adba36e" exitCode=0 Jul 21 13:06:23.213318 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:23.212961 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv" event={"ID":"f9f85524-b092-454a-8d41-baaa24b75709","Type":"ContainerDied","Data":"8ab3cce8498908650152f03ac248f950c0c4eb399234b5b183201c688adba36e"} Jul 21 13:06:23.213318 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:23.213018 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv" event={"ID":"f9f85524-b092-454a-8d41-baaa24b75709","Type":"ContainerStarted","Data":"d6740e05bb4c69741f1c7f70055e04beffde6695abf6263978d4b29b9a80645a"} Jul 21 13:06:24.350796 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:24.350772 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv" Jul 21 13:06:24.429836 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:24.429807 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f85524-b092-454a-8d41-baaa24b75709-maas-api-service-ca\") pod \"f9f85524-b092-454a-8d41-baaa24b75709\" (UID: \"f9f85524-b092-454a-8d41-baaa24b75709\") " Jul 21 13:06:24.430206 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:24.430179 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f85524-b092-454a-8d41-baaa24b75709-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f9f85524-b092-454a-8d41-baaa24b75709" (UID: "f9f85524-b092-454a-8d41-baaa24b75709"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:06:24.530669 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:24.530639 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f85524-b092-454a-8d41-baaa24b75709-maas-api-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:06:25.225250 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:25.225217 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv" Jul 21 13:06:25.225429 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:25.225254 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv" event={"ID":"f9f85524-b092-454a-8d41-baaa24b75709","Type":"ContainerDied","Data":"d6740e05bb4c69741f1c7f70055e04beffde6695abf6263978d4b29b9a80645a"} Jul 21 13:06:25.225429 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:25.225300 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6740e05bb4c69741f1c7f70055e04beffde6695abf6263978d4b29b9a80645a" Jul 21 13:06:37.358808 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:37.358767 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl"] Jul 21 13:06:37.359324 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:37.359016 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" podUID="d0c626e0-f3c5-453d-9582-434b8250d678" containerName="maas-api" containerID="cri-o://d69f48ecedce5d149bf9ef25d09d019df399fb8e4f7462f467dd658971504c16" gracePeriod=30 Jul 21 13:06:37.612734 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:37.612677 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:06:37.753796 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:37.753757 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c626e0-f3c5-453d-9582-434b8250d678-openshift-service-ca\") pod \"d0c626e0-f3c5-453d-9582-434b8250d678\" (UID: \"d0c626e0-f3c5-453d-9582-434b8250d678\") " Jul 21 13:06:37.754023 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:37.753829 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8bth\" (UniqueName: \"kubernetes.io/projected/d0c626e0-f3c5-453d-9582-434b8250d678-kube-api-access-v8bth\") pod \"d0c626e0-f3c5-453d-9582-434b8250d678\" (UID: \"d0c626e0-f3c5-453d-9582-434b8250d678\") " Jul 21 13:06:37.754023 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:37.753853 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0c626e0-f3c5-453d-9582-434b8250d678-maas-api-tls\") pod \"d0c626e0-f3c5-453d-9582-434b8250d678\" (UID: \"d0c626e0-f3c5-453d-9582-434b8250d678\") " Jul 21 13:06:37.754250 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:37.754223 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c626e0-f3c5-453d-9582-434b8250d678-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d0c626e0-f3c5-453d-9582-434b8250d678" (UID: "d0c626e0-f3c5-453d-9582-434b8250d678"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:06:37.755932 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:37.755911 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c626e0-f3c5-453d-9582-434b8250d678-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d0c626e0-f3c5-453d-9582-434b8250d678" (UID: "d0c626e0-f3c5-453d-9582-434b8250d678"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:06:37.756068 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:37.756048 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c626e0-f3c5-453d-9582-434b8250d678-kube-api-access-v8bth" (OuterVolumeSpecName: "kube-api-access-v8bth") pod "d0c626e0-f3c5-453d-9582-434b8250d678" (UID: "d0c626e0-f3c5-453d-9582-434b8250d678"). InnerVolumeSpecName "kube-api-access-v8bth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:06:37.854869 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:37.854831 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c626e0-f3c5-453d-9582-434b8250d678-openshift-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:06:37.854869 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:37.854861 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8bth\" (UniqueName: \"kubernetes.io/projected/d0c626e0-f3c5-453d-9582-434b8250d678-kube-api-access-v8bth\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:06:37.854869 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:37.854872 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0c626e0-f3c5-453d-9582-434b8250d678-maas-api-tls\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:06:38.297081 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:38.297043 2574 generic.go:356] "Generic (PLEG): container finished" podID="d0c626e0-f3c5-453d-9582-434b8250d678" containerID="d69f48ecedce5d149bf9ef25d09d019df399fb8e4f7462f467dd658971504c16" exitCode=0 Jul 21 13:06:38.297279 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:38.297096 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" Jul 21 13:06:38.297279 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:38.297132 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" event={"ID":"d0c626e0-f3c5-453d-9582-434b8250d678","Type":"ContainerDied","Data":"d69f48ecedce5d149bf9ef25d09d019df399fb8e4f7462f467dd658971504c16"} Jul 21 13:06:38.297279 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:38.297184 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl" event={"ID":"d0c626e0-f3c5-453d-9582-434b8250d678","Type":"ContainerDied","Data":"20ad5cc832ce6ad2339af65c19a34d04be32709c10bcc706b19e39e10545eeb1"} Jul 21 13:06:38.297279 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:38.297207 2574 scope.go:117] "RemoveContainer" containerID="d69f48ecedce5d149bf9ef25d09d019df399fb8e4f7462f467dd658971504c16" Jul 21 13:06:38.309484 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:38.309462 2574 scope.go:117] "RemoveContainer" containerID="d69f48ecedce5d149bf9ef25d09d019df399fb8e4f7462f467dd658971504c16" Jul 21 13:06:38.309765 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:06:38.309743 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69f48ecedce5d149bf9ef25d09d019df399fb8e4f7462f467dd658971504c16\": container with ID starting with d69f48ecedce5d149bf9ef25d09d019df399fb8e4f7462f467dd658971504c16 not found: ID does not exist" containerID="d69f48ecedce5d149bf9ef25d09d019df399fb8e4f7462f467dd658971504c16" Jul 21 13:06:38.309841 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:38.309773 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69f48ecedce5d149bf9ef25d09d019df399fb8e4f7462f467dd658971504c16"} err="failed to get container status \"d69f48ecedce5d149bf9ef25d09d019df399fb8e4f7462f467dd658971504c16\": rpc error: code = NotFound desc = could not find container \"d69f48ecedce5d149bf9ef25d09d019df399fb8e4f7462f467dd658971504c16\": container with ID starting with d69f48ecedce5d149bf9ef25d09d019df399fb8e4f7462f467dd658971504c16 not found: ID does not exist" Jul 21 13:06:38.315427 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:38.315401 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl"] Jul 21 13:06:38.319413 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:38.319390 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-69d4e8-6b58d868f5-lbbwl"] Jul 21 13:06:39.924429 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:39.924392 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0c626e0-f3c5-453d-9582-434b8250d678" path="/var/lib/kubelet/pods/d0c626e0-f3c5-453d-9582-434b8250d678/volumes" Jul 21 13:06:41.924502 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:41.924471 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv"] Jul 21 13:06:41.929044 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:41.929013 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69d4e8-tprxv"] Jul 21 13:06:43.924256 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:43.924222 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9f85524-b092-454a-8d41-baaa24b75709" path="/var/lib/kubelet/pods/f9f85524-b092-454a-8d41-baaa24b75709/volumes" Jul 21 13:06:52.364769 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.364733 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v"] Jul 21 13:06:52.365323 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.365306 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c626e0-f3c5-453d-9582-434b8250d678" containerName="maas-api" Jul 21 13:06:52.365369 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.365326 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c626e0-f3c5-453d-9582-434b8250d678" containerName="maas-api" Jul 21 13:06:52.365403 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.365368 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f85524-b092-454a-8d41-baaa24b75709" containerName="revoke-keys" Jul 21 13:06:52.365403 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.365377 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f85524-b092-454a-8d41-baaa24b75709" containerName="revoke-keys" Jul 21 13:06:52.365470 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.365458 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9f85524-b092-454a-8d41-baaa24b75709" containerName="revoke-keys" Jul 21 13:06:52.365505 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.365476 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c626e0-f3c5-453d-9582-434b8250d678" containerName="maas-api" Jul 21 13:06:52.369182 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.369159 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:06:52.371691 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.371663 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-3cf59c-serving-cert\"" Jul 21 13:06:52.378472 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.377995 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v"] Jul 21 13:06:52.381485 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.381459 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91650f8a-8e08-4469-b825-38da2c885f8d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v\" (UID: \"91650f8a-8e08-4469-b825-38da2c885f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:06:52.381597 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.381525 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91650f8a-8e08-4469-b825-38da2c885f8d-maas-api-tls\") pod \"maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v\" (UID: \"91650f8a-8e08-4469-b825-38da2c885f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:06:52.381597 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.381577 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsrqd\" (UniqueName: \"kubernetes.io/projected/91650f8a-8e08-4469-b825-38da2c885f8d-kube-api-access-xsrqd\") pod \"maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v\" (UID: \"91650f8a-8e08-4469-b825-38da2c885f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:06:52.482879 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.482842 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91650f8a-8e08-4469-b825-38da2c885f8d-maas-api-tls\") pod \"maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v\" (UID: \"91650f8a-8e08-4469-b825-38da2c885f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:06:52.483077 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.482886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsrqd\" (UniqueName: \"kubernetes.io/projected/91650f8a-8e08-4469-b825-38da2c885f8d-kube-api-access-xsrqd\") pod \"maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v\" (UID: \"91650f8a-8e08-4469-b825-38da2c885f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:06:52.483077 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.482980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91650f8a-8e08-4469-b825-38da2c885f8d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v\" (UID: \"91650f8a-8e08-4469-b825-38da2c885f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:06:52.483077 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:06:52.483013 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-serving-cert: secret "maas-api-e2e-shared-a-3cf59c-serving-cert" not found Jul 21 13:06:52.483204 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:06:52.483094 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91650f8a-8e08-4469-b825-38da2c885f8d-maas-api-tls podName:91650f8a-8e08-4469-b825-38da2c885f8d nodeName:}" failed. No retries permitted until 2026-07-21 13:06:52.983071747 +0000 UTC m=+2359.633485114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91650f8a-8e08-4469-b825-38da2c885f8d-maas-api-tls") pod "maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" (UID: "91650f8a-8e08-4469-b825-38da2c885f8d") : secret "maas-api-e2e-shared-a-3cf59c-serving-cert" not found Jul 21 13:06:52.483655 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.483636 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91650f8a-8e08-4469-b825-38da2c885f8d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v\" (UID: \"91650f8a-8e08-4469-b825-38da2c885f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:06:52.498255 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.498230 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsrqd\" (UniqueName: \"kubernetes.io/projected/91650f8a-8e08-4469-b825-38da2c885f8d-kube-api-access-xsrqd\") pod \"maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v\" (UID: \"91650f8a-8e08-4469-b825-38da2c885f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:06:52.987519 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.987483 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91650f8a-8e08-4469-b825-38da2c885f8d-maas-api-tls\") pod \"maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v\" (UID: \"91650f8a-8e08-4469-b825-38da2c885f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:06:52.989997 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:52.989974 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91650f8a-8e08-4469-b825-38da2c885f8d-maas-api-tls\") pod \"maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v\" (UID: \"91650f8a-8e08-4469-b825-38da2c885f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:06:53.282616 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:53.282580 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:06:53.443807 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:53.443779 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v"] Jul 21 13:06:53.445386 ip-10-0-138-11 kubenswrapper[2574]: W0721 13:06:53.445357 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91650f8a_8e08_4469_b825_38da2c885f8d.slice/crio-a440a915011bfc702b99e3368128016371212aff737edc0359310707cafd6694 WatchSource:0}: Error finding container a440a915011bfc702b99e3368128016371212aff737edc0359310707cafd6694: Status 404 returned error can't find the container with id a440a915011bfc702b99e3368128016371212aff737edc0359310707cafd6694 Jul 21 13:06:54.398956 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:54.398893 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" event={"ID":"91650f8a-8e08-4469-b825-38da2c885f8d","Type":"ContainerStarted","Data":"a440a915011bfc702b99e3368128016371212aff737edc0359310707cafd6694"} Jul 21 13:06:55.406453 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:55.406414 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" event={"ID":"91650f8a-8e08-4469-b825-38da2c885f8d","Type":"ContainerStarted","Data":"b4d21ad4afff21921835afa452ae864081a69b542d34fa2881730dcfd1f7c78b"} Jul 21 13:06:55.406876 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:55.406568 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:06:55.426848 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:06:55.426796 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" podStartSLOduration=2.305153999 podStartE2EDuration="3.42678242s" totalImagesPullingTime="1.121628421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:06:55.425142321 +0000 UTC m=+2362.075555695" watchObservedRunningTime="2026-07-21 13:06:55.42678242 +0000 UTC m=+2362.077195780" Jul 21 13:07:00.518101 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.518068 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d"] Jul 21 13:07:00.522019 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.521999 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:07:00.524910 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.524887 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-b1b184-serving-cert\"" Jul 21 13:07:00.536370 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.536346 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d"] Jul 21 13:07:00.557339 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.557297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjp4\" (UniqueName: \"kubernetes.io/projected/8511c4e0-4c0e-4885-bb35-2d489ab87f85-kube-api-access-fhjp4\") pod \"maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d\" (UID: \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:07:00.557511 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.557393 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8511c4e0-4c0e-4885-bb35-2d489ab87f85-maas-api-tls\") pod \"maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d\" (UID: \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:07:00.557511 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.557461 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8511c4e0-4c0e-4885-bb35-2d489ab87f85-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d\" (UID: \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:07:00.658770 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.658738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjp4\" (UniqueName: \"kubernetes.io/projected/8511c4e0-4c0e-4885-bb35-2d489ab87f85-kube-api-access-fhjp4\") pod \"maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d\" (UID: \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:07:00.658996 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.658797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8511c4e0-4c0e-4885-bb35-2d489ab87f85-maas-api-tls\") pod \"maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d\" (UID: \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:07:00.658996 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.658865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8511c4e0-4c0e-4885-bb35-2d489ab87f85-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d\" (UID: \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:07:00.659567 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.659538 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8511c4e0-4c0e-4885-bb35-2d489ab87f85-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d\" (UID: \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:07:00.661556 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.661534 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8511c4e0-4c0e-4885-bb35-2d489ab87f85-maas-api-tls\") pod \"maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d\" (UID: \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:07:00.676186 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.676163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjp4\" (UniqueName: \"kubernetes.io/projected/8511c4e0-4c0e-4885-bb35-2d489ab87f85-kube-api-access-fhjp4\") pod \"maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d\" (UID: \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:07:00.831959 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.831858 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:07:00.973187 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:00.973163 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d"] Jul 21 13:07:00.975407 ip-10-0-138-11 kubenswrapper[2574]: W0721 13:07:00.975380 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8511c4e0_4c0e_4885_bb35_2d489ab87f85.slice/crio-55cf02efd4ea0c9d0adc1f9c4383596e8d0ab6c8d669839fec978d19ee7bd207 WatchSource:0}: Error finding container 55cf02efd4ea0c9d0adc1f9c4383596e8d0ab6c8d669839fec978d19ee7bd207: Status 404 returned error can't find the container with id 55cf02efd4ea0c9d0adc1f9c4383596e8d0ab6c8d669839fec978d19ee7bd207 Jul 21 13:07:01.417675 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:01.417649 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:07:01.443098 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:01.443068 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" event={"ID":"8511c4e0-4c0e-4885-bb35-2d489ab87f85","Type":"ContainerStarted","Data":"974125158f4b2fd8a602c0479fcaaa347ef6d2590808b16adeeb8edf5dba9c0e"} Jul 21 13:07:01.443269 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:01.443126 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" event={"ID":"8511c4e0-4c0e-4885-bb35-2d489ab87f85","Type":"ContainerStarted","Data":"55cf02efd4ea0c9d0adc1f9c4383596e8d0ab6c8d669839fec978d19ee7bd207"} Jul 21 13:07:01.443269 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:01.443234 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:07:01.474635 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:01.474574 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" podStartSLOduration=1.119696565 podStartE2EDuration="1.474550924s" totalImagesPullingTime="354.854359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:07:01.473388505 +0000 UTC m=+2368.123801867" watchObservedRunningTime="2026-07-21 13:07:01.474550924 +0000 UTC m=+2368.124964287" Jul 21 13:07:07.455271 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:07.455240 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:07:34.060874 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:34.060848 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 13:07:34.066681 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:34.066658 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 13:07:34.076274 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:34.076254 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 13:07:34.082463 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:34.082445 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 13:07:34.247475 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:07:34.247444 2574 scope.go:117] "RemoveContainer" containerID="096d57251032660fa54f622e75c2a3da738fa3ce5b746ef75e415789db13fd5a" Jul 21 13:11:34.297854 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:11:34.297815 2574 scope.go:117] "RemoveContainer" containerID="ce152725dede81c2f39a9479339ac4fcea1d4cb5d39f995dc300e95a122bb62c" Jul 21 13:12:34.107387 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:12:34.107307 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 13:12:34.113253 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:12:34.113231 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 13:12:34.122772 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:12:34.122752 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 13:12:34.128353 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:12:34.128337 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhsp9_10f19045-4a86-44fa-9e06-60030a2120ca/ovn-acl-logging/0.log" Jul 21 13:12:34.321118 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:12:34.321084 2574 scope.go:117] "RemoveContainer" containerID="8ab3cce8498908650152f03ac248f950c0c4eb399234b5b183201c688adba36e" Jul 21 13:12:34.329824 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:12:34.329804 2574 scope.go:117] "RemoveContainer" containerID="7344aa46c0ade8e23026d9b44e5378f6e42269c160eac56eefbc2730702e7124" Jul 21 13:14:34.401107 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.401064 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v"] Jul 21 13:14:34.401684 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.401391 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" podUID="91650f8a-8e08-4469-b825-38da2c885f8d" containerName="maas-api" containerID="cri-o://b4d21ad4afff21921835afa452ae864081a69b542d34fa2881730dcfd1f7c78b" gracePeriod=30 Jul 21 13:14:34.660966 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.660877 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:14:34.677693 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.677653 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91650f8a-8e08-4469-b825-38da2c885f8d-openshift-service-ca\") pod \"91650f8a-8e08-4469-b825-38da2c885f8d\" (UID: \"91650f8a-8e08-4469-b825-38da2c885f8d\") " Jul 21 13:14:34.677861 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.677730 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91650f8a-8e08-4469-b825-38da2c885f8d-maas-api-tls\") pod \"91650f8a-8e08-4469-b825-38da2c885f8d\" (UID: \"91650f8a-8e08-4469-b825-38da2c885f8d\") " Jul 21 13:14:34.677861 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.677799 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsrqd\" (UniqueName: \"kubernetes.io/projected/91650f8a-8e08-4469-b825-38da2c885f8d-kube-api-access-xsrqd\") pod \"91650f8a-8e08-4469-b825-38da2c885f8d\" (UID: \"91650f8a-8e08-4469-b825-38da2c885f8d\") " Jul 21 13:14:34.678516 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.678324 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91650f8a-8e08-4469-b825-38da2c885f8d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91650f8a-8e08-4469-b825-38da2c885f8d" (UID: "91650f8a-8e08-4469-b825-38da2c885f8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:14:34.680485 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.680422 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91650f8a-8e08-4469-b825-38da2c885f8d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91650f8a-8e08-4469-b825-38da2c885f8d" (UID: "91650f8a-8e08-4469-b825-38da2c885f8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:14:34.681308 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.681272 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91650f8a-8e08-4469-b825-38da2c885f8d-kube-api-access-xsrqd" (OuterVolumeSpecName: "kube-api-access-xsrqd") pod "91650f8a-8e08-4469-b825-38da2c885f8d" (UID: "91650f8a-8e08-4469-b825-38da2c885f8d"). InnerVolumeSpecName "kube-api-access-xsrqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:14:34.779506 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.779469 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsrqd\" (UniqueName: \"kubernetes.io/projected/91650f8a-8e08-4469-b825-38da2c885f8d-kube-api-access-xsrqd\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:14:34.779506 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.779502 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91650f8a-8e08-4469-b825-38da2c885f8d-openshift-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:14:34.779506 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.779512 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91650f8a-8e08-4469-b825-38da2c885f8d-maas-api-tls\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:14:34.922995 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.922883 2574 generic.go:356] "Generic (PLEG): container finished" podID="91650f8a-8e08-4469-b825-38da2c885f8d" containerID="b4d21ad4afff21921835afa452ae864081a69b542d34fa2881730dcfd1f7c78b" exitCode=0 Jul 21 13:14:34.922995 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.922959 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" Jul 21 13:14:34.922995 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.922981 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" event={"ID":"91650f8a-8e08-4469-b825-38da2c885f8d","Type":"ContainerDied","Data":"b4d21ad4afff21921835afa452ae864081a69b542d34fa2881730dcfd1f7c78b"} Jul 21 13:14:34.923243 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.923034 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v" event={"ID":"91650f8a-8e08-4469-b825-38da2c885f8d","Type":"ContainerDied","Data":"a440a915011bfc702b99e3368128016371212aff737edc0359310707cafd6694"} Jul 21 13:14:34.923243 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.923080 2574 scope.go:117] "RemoveContainer" containerID="b4d21ad4afff21921835afa452ae864081a69b542d34fa2881730dcfd1f7c78b" Jul 21 13:14:34.933200 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.933181 2574 scope.go:117] "RemoveContainer" containerID="b4d21ad4afff21921835afa452ae864081a69b542d34fa2881730dcfd1f7c78b" Jul 21 13:14:34.933508 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:14:34.933478 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d21ad4afff21921835afa452ae864081a69b542d34fa2881730dcfd1f7c78b\": container with ID starting with b4d21ad4afff21921835afa452ae864081a69b542d34fa2881730dcfd1f7c78b not found: ID does not exist" containerID="b4d21ad4afff21921835afa452ae864081a69b542d34fa2881730dcfd1f7c78b" Jul 21 13:14:34.933612 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.933519 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d21ad4afff21921835afa452ae864081a69b542d34fa2881730dcfd1f7c78b"} err="failed to get container status \"b4d21ad4afff21921835afa452ae864081a69b542d34fa2881730dcfd1f7c78b\": rpc error: code = NotFound desc = could not find container \"b4d21ad4afff21921835afa452ae864081a69b542d34fa2881730dcfd1f7c78b\": container with ID starting with b4d21ad4afff21921835afa452ae864081a69b542d34fa2881730dcfd1f7c78b not found: ID does not exist" Jul 21 13:14:34.947917 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.947876 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v"] Jul 21 13:14:34.951317 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:34.951287 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3cf59c-6cf449897d-cwh8v"] Jul 21 13:14:35.924659 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:35.924622 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91650f8a-8e08-4469-b825-38da2c885f8d" path="/var/lib/kubelet/pods/91650f8a-8e08-4469-b825-38da2c885f8d/volumes" Jul 21 13:14:46.144767 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:46.144729 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh"] Jul 21 13:14:46.145329 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:46.145309 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91650f8a-8e08-4469-b825-38da2c885f8d" containerName="maas-api" Jul 21 13:14:46.145418 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:46.145332 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="91650f8a-8e08-4469-b825-38da2c885f8d" containerName="maas-api" Jul 21 13:14:46.145471 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:46.145438 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="91650f8a-8e08-4469-b825-38da2c885f8d" containerName="maas-api" Jul 21 13:14:46.148440 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:46.148419 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh" Jul 21 13:14:46.151404 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:46.151375 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jq8px\"" Jul 21 13:14:46.156882 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:46.156853 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh"] Jul 21 13:14:46.185118 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:46.185085 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd5a6f3-95e6-434a-8be5-46c19dcf8650-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh\" (UID: \"2dd5a6f3-95e6-434a-8be5-46c19dcf8650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh" Jul 21 13:14:46.286123 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:46.286078 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd5a6f3-95e6-434a-8be5-46c19dcf8650-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh\" (UID: \"2dd5a6f3-95e6-434a-8be5-46c19dcf8650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh" Jul 21 13:14:46.286757 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:46.286732 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd5a6f3-95e6-434a-8be5-46c19dcf8650-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh\" (UID: \"2dd5a6f3-95e6-434a-8be5-46c19dcf8650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh" Jul 21 13:14:46.464476 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:46.464384 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh" Jul 21 13:14:46.608834 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:46.608801 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh"] Jul 21 13:14:46.609983 ip-10-0-138-11 kubenswrapper[2574]: W0721 13:14:46.609920 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd5a6f3_95e6_434a_8be5_46c19dcf8650.slice/crio-648d7ee3d856bc4c4a3f991a51659ce8441e05cb60ea07b8d53006e58eedb180 WatchSource:0}: Error finding container 648d7ee3d856bc4c4a3f991a51659ce8441e05cb60ea07b8d53006e58eedb180: Status 404 returned error can't find the container with id 648d7ee3d856bc4c4a3f991a51659ce8441e05cb60ea07b8d53006e58eedb180 Jul 21 13:14:46.997672 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:46.997630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh" event={"ID":"2dd5a6f3-95e6-434a-8be5-46c19dcf8650","Type":"ContainerStarted","Data":"4a21822b285e81398b4e48ad436e93887d5c5ade83ca66c78e4887589d457fe1"} Jul 21 13:14:46.997827 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:46.997718 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh" event={"ID":"2dd5a6f3-95e6-434a-8be5-46c19dcf8650","Type":"ContainerStarted","Data":"648d7ee3d856bc4c4a3f991a51659ce8441e05cb60ea07b8d53006e58eedb180"} Jul 21 13:14:47.026199 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:47.026091 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh" podStartSLOduration=1.026073654 podStartE2EDuration="1.026073654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:14:47.02491103 +0000 UTC m=+2833.675324393" watchObservedRunningTime="2026-07-21 13:14:47.026073654 +0000 UTC m=+2833.676487024" Jul 21 13:14:48.004551 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:48.004516 2574 generic.go:356] "Generic (PLEG): container finished" podID="2dd5a6f3-95e6-434a-8be5-46c19dcf8650" containerID="4a21822b285e81398b4e48ad436e93887d5c5ade83ca66c78e4887589d457fe1" exitCode=0 Jul 21 13:14:48.004978 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:48.004603 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh" event={"ID":"2dd5a6f3-95e6-434a-8be5-46c19dcf8650","Type":"ContainerDied","Data":"4a21822b285e81398b4e48ad436e93887d5c5ade83ca66c78e4887589d457fe1"} Jul 21 13:14:49.146444 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:49.146416 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh" Jul 21 13:14:49.219428 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:49.219373 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd5a6f3-95e6-434a-8be5-46c19dcf8650-maas-api-service-ca\") pod \"2dd5a6f3-95e6-434a-8be5-46c19dcf8650\" (UID: \"2dd5a6f3-95e6-434a-8be5-46c19dcf8650\") " Jul 21 13:14:49.219797 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:49.219775 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd5a6f3-95e6-434a-8be5-46c19dcf8650-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2dd5a6f3-95e6-434a-8be5-46c19dcf8650" (UID: "2dd5a6f3-95e6-434a-8be5-46c19dcf8650"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:14:49.321054 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:49.320966 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd5a6f3-95e6-434a-8be5-46c19dcf8650-maas-api-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:14:50.016262 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:50.016233 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh" Jul 21 13:14:50.016431 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:50.016268 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh" event={"ID":"2dd5a6f3-95e6-434a-8be5-46c19dcf8650","Type":"ContainerDied","Data":"648d7ee3d856bc4c4a3f991a51659ce8441e05cb60ea07b8d53006e58eedb180"} Jul 21 13:14:50.016431 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:50.016316 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="648d7ee3d856bc4c4a3f991a51659ce8441e05cb60ea07b8d53006e58eedb180" Jul 21 13:14:56.575102 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:56.575066 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d"] Jul 21 13:14:56.575755 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:56.575721 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" podUID="8511c4e0-4c0e-4885-bb35-2d489ab87f85" containerName="maas-api" containerID="cri-o://974125158f4b2fd8a602c0479fcaaa347ef6d2590808b16adeeb8edf5dba9c0e" gracePeriod=30 Jul 21 13:14:56.829385 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:56.829313 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:14:56.989403 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:56.989368 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8511c4e0-4c0e-4885-bb35-2d489ab87f85-maas-api-tls\") pod \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\" (UID: \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\") " Jul 21 13:14:56.989583 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:56.989452 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhjp4\" (UniqueName: \"kubernetes.io/projected/8511c4e0-4c0e-4885-bb35-2d489ab87f85-kube-api-access-fhjp4\") pod \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\" (UID: \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\") " Jul 21 13:14:56.989583 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:56.989551 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8511c4e0-4c0e-4885-bb35-2d489ab87f85-openshift-service-ca\") pod \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\" (UID: \"8511c4e0-4c0e-4885-bb35-2d489ab87f85\") " Jul 21 13:14:56.989923 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:56.989894 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8511c4e0-4c0e-4885-bb35-2d489ab87f85-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8511c4e0-4c0e-4885-bb35-2d489ab87f85" (UID: "8511c4e0-4c0e-4885-bb35-2d489ab87f85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 13:14:56.991457 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:56.991431 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8511c4e0-4c0e-4885-bb35-2d489ab87f85-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8511c4e0-4c0e-4885-bb35-2d489ab87f85" (UID: "8511c4e0-4c0e-4885-bb35-2d489ab87f85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:14:56.991564 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:56.991540 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8511c4e0-4c0e-4885-bb35-2d489ab87f85-kube-api-access-fhjp4" (OuterVolumeSpecName: "kube-api-access-fhjp4") pod "8511c4e0-4c0e-4885-bb35-2d489ab87f85" (UID: "8511c4e0-4c0e-4885-bb35-2d489ab87f85"). InnerVolumeSpecName "kube-api-access-fhjp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:14:57.055878 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:57.055835 2574 generic.go:356] "Generic (PLEG): container finished" podID="8511c4e0-4c0e-4885-bb35-2d489ab87f85" containerID="974125158f4b2fd8a602c0479fcaaa347ef6d2590808b16adeeb8edf5dba9c0e" exitCode=0 Jul 21 13:14:57.056069 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:57.055891 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" event={"ID":"8511c4e0-4c0e-4885-bb35-2d489ab87f85","Type":"ContainerDied","Data":"974125158f4b2fd8a602c0479fcaaa347ef6d2590808b16adeeb8edf5dba9c0e"} Jul 21 13:14:57.056069 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:57.055935 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" event={"ID":"8511c4e0-4c0e-4885-bb35-2d489ab87f85","Type":"ContainerDied","Data":"55cf02efd4ea0c9d0adc1f9c4383596e8d0ab6c8d669839fec978d19ee7bd207"} Jul 21 13:14:57.056069 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:57.055892 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d" Jul 21 13:14:57.056069 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:57.055996 2574 scope.go:117] "RemoveContainer" containerID="974125158f4b2fd8a602c0479fcaaa347ef6d2590808b16adeeb8edf5dba9c0e" Jul 21 13:14:57.066137 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:57.066118 2574 scope.go:117] "RemoveContainer" containerID="974125158f4b2fd8a602c0479fcaaa347ef6d2590808b16adeeb8edf5dba9c0e" Jul 21 13:14:57.066418 ip-10-0-138-11 kubenswrapper[2574]: E0721 13:14:57.066398 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974125158f4b2fd8a602c0479fcaaa347ef6d2590808b16adeeb8edf5dba9c0e\": container with ID starting with 974125158f4b2fd8a602c0479fcaaa347ef6d2590808b16adeeb8edf5dba9c0e not found: ID does not exist" containerID="974125158f4b2fd8a602c0479fcaaa347ef6d2590808b16adeeb8edf5dba9c0e" Jul 21 13:14:57.066490 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:57.066432 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974125158f4b2fd8a602c0479fcaaa347ef6d2590808b16adeeb8edf5dba9c0e"} err="failed to get container status \"974125158f4b2fd8a602c0479fcaaa347ef6d2590808b16adeeb8edf5dba9c0e\": rpc error: code = NotFound desc = could not find container \"974125158f4b2fd8a602c0479fcaaa347ef6d2590808b16adeeb8edf5dba9c0e\": container with ID starting with 974125158f4b2fd8a602c0479fcaaa347ef6d2590808b16adeeb8edf5dba9c0e not found: ID does not exist" Jul 21 13:14:57.079503 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:57.079431 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d"] Jul 21 13:14:57.085421 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:57.085390 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b1b184-bbd9cfc8b-j6w2d"] Jul 21 13:14:57.091274 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:57.091250 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8511c4e0-4c0e-4885-bb35-2d489ab87f85-maas-api-tls\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:14:57.091360 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:57.091277 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhjp4\" (UniqueName: \"kubernetes.io/projected/8511c4e0-4c0e-4885-bb35-2d489ab87f85-kube-api-access-fhjp4\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:14:57.091360 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:57.091289 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8511c4e0-4c0e-4885-bb35-2d489ab87f85-openshift-service-ca\") on node \"ip-10-0-138-11.ec2.internal\" DevicePath \"\"" Jul 21 13:14:57.927315 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:14:57.927279 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8511c4e0-4c0e-4885-bb35-2d489ab87f85" path="/var/lib/kubelet/pods/8511c4e0-4c0e-4885-bb35-2d489ab87f85/volumes" Jul 21 13:15:01.237781 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:01.237738 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh"] Jul 21 13:15:01.241306 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:01.241278 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b1b184-m9gjh"] Jul 21 13:15:01.924066 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:01.924023 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd5a6f3-95e6-434a-8be5-46c19dcf8650" path="/var/lib/kubelet/pods/2dd5a6f3-95e6-434a-8be5-46c19dcf8650/volumes" Jul 21 13:15:14.550425 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:14.550391 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68fb8f6cb6-w9hbr_771bcead-5e3c-432f-b402-84affce9c8b1/manager/0.log" Jul 21 13:15:15.072349 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:15.072321 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cb6668c8b-tcfj9_69ca45fa-f0f8-4f04-acbe-580377f55252/copy-manifests/0.log" Jul 21 13:15:15.080377 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:15.080351 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cb6668c8b-tcfj9_69ca45fa-f0f8-4f04-acbe-580377f55252/manager/0.log" Jul 21 13:15:15.609019 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:15.608969 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5cf648dd67-hqdgj_cb66477c-21d2-47c5-a8ab-86a4a0a3bb35/maas-api/0.log" Jul 21 13:15:15.742827 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:15.742794 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29743965-fzt6n_3525e52d-141a-4c8c-af85-edc4be0d8bc5/cleanup/0.log" Jul 21 13:15:16.105159 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:16.105127 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-rbzxk_d6ff522d-fdc1-4d74-9ec0-e06665f17706/postgres/0.log" Jul 21 13:15:16.934024 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:16.933992 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f_5644f7de-07a8-4d2b-829b-d52210da47f5/util/0.log" Jul 21 13:15:16.941224 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:16.941194 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f_5644f7de-07a8-4d2b-829b-d52210da47f5/pull/0.log" Jul 21 13:15:16.947890 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:16.947863 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f_5644f7de-07a8-4d2b-829b-d52210da47f5/extract/0.log" Jul 21 13:15:17.065087 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:17.065056 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5_76b9de7e-f20d-415d-861d-ff5aff65c788/pull/0.log" Jul 21 13:15:17.072159 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:17.072128 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5_76b9de7e-f20d-415d-861d-ff5aff65c788/extract/0.log" Jul 21 13:15:17.078710 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:17.078685 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5_76b9de7e-f20d-415d-861d-ff5aff65c788/util/0.log" Jul 21 13:15:17.208350 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:17.208263 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4_89ddd4ed-1f36-4839-a9a2-edb28d549c98/extract/0.log" Jul 21 13:15:17.215858 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:17.215829 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4_89ddd4ed-1f36-4839-a9a2-edb28d549c98/util/0.log" Jul 21 13:15:17.223469 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:17.223446 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4_89ddd4ed-1f36-4839-a9a2-edb28d549c98/pull/0.log" Jul 21 13:15:17.358685 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:17.358645 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5_54dd00c7-7600-40e0-9053-4f151bea65cd/pull/0.log" Jul 21 13:15:17.365651 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:17.365625 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5_54dd00c7-7600-40e0-9053-4f151bea65cd/extract/0.log" Jul 21 13:15:17.372124 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:17.372087 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5_54dd00c7-7600-40e0-9053-4f151bea65cd/util/0.log" Jul 21 13:15:18.901381 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:18.901348 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vnng4_2e72ec98-c294-4082-b7f5-35226069ed1a/discovery/0.log" Jul 21 13:15:19.396368 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:19.396335 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-d82c7_8eb5c4cd-d237-4a89-b5a0-ee640f3afa27/payload-pre-processing/0.log" Jul 21 13:15:19.517208 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:19.517169 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tczgm_612b3adb-d198-4182-a98e-78261ab1fa85/payload-processing/0.log" Jul 21 13:15:20.103763 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:20.103733 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-rn4xv_391cd2e1-763f-4b3e-ab2d-230ea901d9c5/main/0.log" Jul 21 13:15:20.356138 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:20.356044 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftjmkh_a2c20a86-3a9e-461c-b51a-900d63e1ab97/main/0.log" Jul 21 13:15:29.801461 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:29.801376 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rz4f4_b0be2da5-e2d6-4bae-bdcc-c65eb877fa03/global-pull-secret-syncer/0.log" Jul 21 13:15:29.937218 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:29.937182 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x9n6h_d20b63ed-d383-4d87-b716-99961b0a4a57/konnectivity-agent/0.log" Jul 21 13:15:29.996518 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:29.996485 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-11.ec2.internal_ebf654d695d98b916ddecbfb68313be1/haproxy/0.log" Jul 21 13:15:34.148819 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:34.148792 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f_5644f7de-07a8-4d2b-829b-d52210da47f5/extract/0.log" Jul 21 13:15:34.172981 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:34.172953 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f_5644f7de-07a8-4d2b-829b-d52210da47f5/util/0.log" Jul 21 13:15:34.196967 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:34.196923 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759twt2f_5644f7de-07a8-4d2b-829b-d52210da47f5/pull/0.log" Jul 21 13:15:34.235458 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:34.235430 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5_76b9de7e-f20d-415d-861d-ff5aff65c788/extract/0.log" Jul 21 13:15:34.258160 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:34.258134 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5_76b9de7e-f20d-415d-861d-ff5aff65c788/util/0.log" Jul 21 13:15:34.287766 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:34.287734 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rnsd5_76b9de7e-f20d-415d-861d-ff5aff65c788/pull/0.log" Jul 21 13:15:34.315441 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:34.315412 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4_89ddd4ed-1f36-4839-a9a2-edb28d549c98/extract/0.log" Jul 21 13:15:34.338487 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:34.338453 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4_89ddd4ed-1f36-4839-a9a2-edb28d549c98/util/0.log" Jul 21 13:15:34.362268 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:34.362239 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kzps4_89ddd4ed-1f36-4839-a9a2-edb28d549c98/pull/0.log" Jul 21 13:15:34.400992 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:34.400960 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5_54dd00c7-7600-40e0-9053-4f151bea65cd/extract/0.log" Jul 21 13:15:34.426134 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:34.426107 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5_54dd00c7-7600-40e0-9053-4f151bea65cd/util/0.log" Jul 21 13:15:34.453205 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:34.453174 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16f8k5_54dd00c7-7600-40e0-9053-4f151bea65cd/pull/0.log" Jul 21 13:15:37.265390 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:37.265355 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-qrpqm_37341abd-787c-4e7c-a734-514a6a2ebaa0/cluster-monitoring-operator/0.log" Jul 21 13:15:37.410803 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:37.410771 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-rcqcr_8868a4a9-c37b-45dd-b569-11b1dde7848f/monitoring-plugin/0.log" Jul 21 13:15:37.626175 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:37.626065 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qq577_0a03eece-1a62-4928-b8da-4c54e6f01153/node-exporter/0.log" Jul 21 13:15:37.676906 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:37.676878 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qq577_0a03eece-1a62-4928-b8da-4c54e6f01153/kube-rbac-proxy/0.log" Jul 21 13:15:37.706466 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:37.706442 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qq577_0a03eece-1a62-4928-b8da-4c54e6f01153/init-textfile/0.log" Jul 21 13:15:38.311123 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:38.311065 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-865947956c-gb2lf_f35cbae9-4a0a-4250-8a0c-e205231fc564/telemeter-client/0.log" Jul 21 13:15:38.353798 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:38.353768 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-865947956c-gb2lf_f35cbae9-4a0a-4250-8a0c-e205231fc564/reload/0.log" Jul 21 13:15:38.398033 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:38.397999 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-865947956c-gb2lf_f35cbae9-4a0a-4250-8a0c-e205231fc564/kube-rbac-proxy/0.log" Jul 21 13:15:38.441120 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:38.441091 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d76cc7d4-22llw_be3d825d-876a-4d3e-ab9e-5a988f92b3dd/thanos-query/0.log" Jul 21 13:15:38.485150 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:38.485114 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d76cc7d4-22llw_be3d825d-876a-4d3e-ab9e-5a988f92b3dd/kube-rbac-proxy-web/0.log" Jul 21 13:15:38.517870 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:38.517838 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d76cc7d4-22llw_be3d825d-876a-4d3e-ab9e-5a988f92b3dd/kube-rbac-proxy/0.log" Jul 21 13:15:38.557775 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:38.557744 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d76cc7d4-22llw_be3d825d-876a-4d3e-ab9e-5a988f92b3dd/prom-label-proxy/0.log" Jul 21 13:15:38.598865 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:38.598770 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d76cc7d4-22llw_be3d825d-876a-4d3e-ab9e-5a988f92b3dd/kube-rbac-proxy-rules/0.log" Jul 21 13:15:38.636881 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:38.636856 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d76cc7d4-22llw_be3d825d-876a-4d3e-ab9e-5a988f92b3dd/kube-rbac-proxy-metrics/0.log" Jul 21 13:15:39.400890 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.400850 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg"] Jul 21 13:15:39.401431 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.401406 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8511c4e0-4c0e-4885-bb35-2d489ab87f85" containerName="maas-api" Jul 21 13:15:39.401675 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.401434 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8511c4e0-4c0e-4885-bb35-2d489ab87f85" containerName="maas-api" Jul 21 13:15:39.401675 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.401464 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd5a6f3-95e6-434a-8be5-46c19dcf8650" containerName="revoke-keys" Jul 21 13:15:39.401675 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.401476 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd5a6f3-95e6-434a-8be5-46c19dcf8650" containerName="revoke-keys" Jul 21 13:15:39.401675 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.401612 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd5a6f3-95e6-434a-8be5-46c19dcf8650" containerName="revoke-keys" Jul 21 13:15:39.401675 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.401625 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8511c4e0-4c0e-4885-bb35-2d489ab87f85" containerName="maas-api" Jul 21 13:15:39.406269 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.406250 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.409184 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.409163 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lpgz9\"/\"kube-root-ca.crt\"" Jul 21 13:15:39.410152 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.410129 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lpgz9\"/\"default-dockercfg-4r69j\"" Jul 21 13:15:39.410152 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.410143 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lpgz9\"/\"openshift-service-ca.crt\"" Jul 21 13:15:39.415349 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.415328 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg"] Jul 21 13:15:39.488665 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.488623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cd902db-b92b-4dd4-82e7-181616d0f338-lib-modules\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.488863 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.488680 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cd902db-b92b-4dd4-82e7-181616d0f338-sys\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.488863 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.488751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwkw6\" (UniqueName: \"kubernetes.io/projected/5cd902db-b92b-4dd4-82e7-181616d0f338-kube-api-access-wwkw6\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.488863 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.488793 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5cd902db-b92b-4dd4-82e7-181616d0f338-podres\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.488863 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.488855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5cd902db-b92b-4dd4-82e7-181616d0f338-proc\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.590214 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.590176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cd902db-b92b-4dd4-82e7-181616d0f338-sys\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.590387 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.590222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwkw6\" (UniqueName: \"kubernetes.io/projected/5cd902db-b92b-4dd4-82e7-181616d0f338-kube-api-access-wwkw6\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.590387 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.590252 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5cd902db-b92b-4dd4-82e7-181616d0f338-podres\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.590387 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.590304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5cd902db-b92b-4dd4-82e7-181616d0f338-proc\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.590387 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.590313 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cd902db-b92b-4dd4-82e7-181616d0f338-sys\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.590387 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.590383 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5cd902db-b92b-4dd4-82e7-181616d0f338-proc\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.590638 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.590414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cd902db-b92b-4dd4-82e7-181616d0f338-lib-modules\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.590638 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.590445 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5cd902db-b92b-4dd4-82e7-181616d0f338-podres\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.590638 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.590545 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cd902db-b92b-4dd4-82e7-181616d0f338-lib-modules\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.598933 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.598899 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwkw6\" (UniqueName: \"kubernetes.io/projected/5cd902db-b92b-4dd4-82e7-181616d0f338-kube-api-access-wwkw6\") pod \"perf-node-gather-daemonset-2wcvg\" (UID: \"5cd902db-b92b-4dd4-82e7-181616d0f338\") " pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.717181 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.717077 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:39.879894 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:39.879864 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg"] Jul 21 13:15:39.881161 ip-10-0-138-11 kubenswrapper[2574]: W0721 13:15:39.881138 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5cd902db_b92b_4dd4_82e7_181616d0f338.slice/crio-4bea34d9644698f383dcbcfcbcd232949936d7166d6b08ad0b842bab52283c6f WatchSource:0}: Error finding container 4bea34d9644698f383dcbcfcbcd232949936d7166d6b08ad0b842bab52283c6f: Status 404 returned error can't find the container with id 4bea34d9644698f383dcbcfcbcd232949936d7166d6b08ad0b842bab52283c6f Jul 21 13:15:40.298542 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:40.298503 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" event={"ID":"5cd902db-b92b-4dd4-82e7-181616d0f338","Type":"ContainerStarted","Data":"3ac9b2d7a71690156b75f2a5aa7bc53e13069bc31329581ce85cd54540ae2479"} Jul 21 13:15:40.298733 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:40.298561 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" event={"ID":"5cd902db-b92b-4dd4-82e7-181616d0f338","Type":"ContainerStarted","Data":"4bea34d9644698f383dcbcfcbcd232949936d7166d6b08ad0b842bab52283c6f"} Jul 21 13:15:40.298733 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:40.298583 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:40.317074 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:40.317017 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" podStartSLOduration=1.316974931 podStartE2EDuration="1.316974931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 13:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:15:40.316365654 +0000 UTC m=+2886.966779016" watchObservedRunningTime="2026-07-21 13:15:40.316974931 +0000 UTC m=+2886.967388293" Jul 21 13:15:40.476045 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:40.476016 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/1.log" Jul 21 13:15:40.486444 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:40.486407 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-bc4k4_e4321016-2119-4390-8984-e74a3d4da956/console-operator/2.log" Jul 21 13:15:41.600792 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:41.600748 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-n6plj_4955ba99-c918-417c-8e62-85e8298bf82c/volume-data-source-validator/0.log" Jul 21 13:15:42.663989 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:42.663958 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c4jdq_9de2e733-a5ab-429f-972e-b375324a5f82/dns/0.log" Jul 21 13:15:42.685953 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:42.685908 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c4jdq_9de2e733-a5ab-429f-972e-b375324a5f82/kube-rbac-proxy/0.log" Jul 21 13:15:42.787304 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:42.787272 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gqmm7_09deecb4-7e46-42cf-804d-634310995a12/dns-node-resolver/0.log" Jul 21 13:15:43.361352 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:43.361315 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d946d445-xgmq9_01f4155a-f514-4452-b5c7-23bc7db245f0/registry/0.log" Jul 21 13:15:43.411240 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:43.411209 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lxclp_795fc54b-5129-4ef3-8fba-7aab06517e1f/node-ca/0.log" Jul 21 13:15:44.579561 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:44.579526 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vnng4_2e72ec98-c294-4082-b7f5-35226069ed1a/discovery/0.log" Jul 21 13:15:44.830008 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:44.829907 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-d82c7_8eb5c4cd-d237-4a89-b5a0-ee640f3afa27/payload-pre-processing/0.log" Jul 21 13:15:44.865956 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:44.865912 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tczgm_612b3adb-d198-4182-a98e-78261ab1fa85/payload-processing/0.log" Jul 21 13:15:45.604556 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:45.604523 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t6fkr_493eec89-2b07-4c39-a989-feea5c5a9a1b/serve-healthcheck-canary/0.log" Jul 21 13:15:46.086578 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:46.086535 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-59v4g_4d964404-b5ad-45f6-90c0-e70bb3986442/insights-operator/0.log" Jul 21 13:15:46.087435 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:46.087408 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-59v4g_4d964404-b5ad-45f6-90c0-e70bb3986442/insights-operator/1.log" Jul 21 13:15:46.187870 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:46.187839 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9p2tq_0e3e9900-040e-467b-91ce-18d7406625ee/kube-rbac-proxy/0.log" Jul 21 13:15:46.210976 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:46.210933 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9p2tq_0e3e9900-040e-467b-91ce-18d7406625ee/exporter/0.log" Jul 21 13:15:46.235060 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:46.235022 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9p2tq_0e3e9900-040e-467b-91ce-18d7406625ee/extractor/0.log" Jul 21 13:15:46.313505 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:46.313473 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lpgz9/perf-node-gather-daemonset-2wcvg" Jul 21 13:15:49.137866 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:49.137830 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68fb8f6cb6-w9hbr_771bcead-5e3c-432f-b402-84affce9c8b1/manager/0.log" Jul 21 13:15:49.389783 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:49.389755 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cb6668c8b-tcfj9_69ca45fa-f0f8-4f04-acbe-580377f55252/manager/0.log" Jul 21 13:15:49.414511 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:49.414487 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cb6668c8b-tcfj9_69ca45fa-f0f8-4f04-acbe-580377f55252/copy-manifests/0.log" Jul 21 13:15:50.091650 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:50.091620 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vnng4_2e72ec98-c294-4082-b7f5-35226069ed1a/discovery/0.log" Jul 21 13:15:50.203497 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:50.203459 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-d82c7_8eb5c4cd-d237-4a89-b5a0-ee640f3afa27/payload-pre-processing/0.log" Jul 21 13:15:50.226699 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:50.226666 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tczgm_612b3adb-d198-4182-a98e-78261ab1fa85/payload-processing/0.log" Jul 21 13:15:50.632249 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:50.632216 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c45b47d79-grmds_85f66440-b3c8-4597-b9ef-0efa9acaaeec/manager/0.log" Jul 21 13:15:51.699153 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:51.699121 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vnng4_2e72ec98-c294-4082-b7f5-35226069ed1a/discovery/0.log" Jul 21 13:15:51.807402 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:51.807377 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-d82c7_8eb5c4cd-d237-4a89-b5a0-ee640f3afa27/payload-pre-processing/0.log" Jul 21 13:15:51.832909 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:51.832884 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tczgm_612b3adb-d198-4182-a98e-78261ab1fa85/payload-processing/0.log" Jul 21 13:15:55.884137 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:55.884052 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-w5n22_8947a59a-697b-41e4-9ecc-dfd60f9eab33/kube-storage-version-migrator-operator/1.log" Jul 21 13:15:55.885520 ip-10-0-138-11 kubenswrapper[2574]: I0721 13:15:55.885493 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-w5n22_8947a59a-697b-41e4-9ecc-dfd60f9eab33/kube-storage-version-migrator-operator/0.log"